Fixing production without break-glass

Last weekend one of our production nodes started flapping. A bulk job blew out the heap on confighub server which crashed the underlying node, not just the pod. The pod had a tiny memory request and no limit, so nothing contained it.

An agent started diagnosing as soon as we got the ops alert. The memory issue was apparent and the fix required a config change to production.

This is where most teams break glass: To get the fix in quickly they reach for kubectl edit or an equivalent tool. In our case, we didn't have to because configuration as data offers 3 key advantages during an incident:

The path to production is short. It takes just seconds to propagate config changes, even with full validation, gating and audit trail.
A change can be scoped to exactly the one deployment that is broken, no blast radius risk.
Every field is editable, including the one you need to change right now even when it wasn't planned for.

Let's look at this in detail.

The problem with breaking the glass

Break-glass access exists because the paved road is usually too slow and too narrow for an incident. If your config pipeline takes twenty minutes of CI to reach the cluster, or your platform abstraction only exposes the fields somebody thought should be changeable, then editing production directly is the only move left when the pager goes off.

But it's a bad move for several reasons:

The change isn't recorded anywhere your team looks. The next person to debug the service is debugging a cluster that no longer matches its declared configuration.
It doesn't survive. If you run GitOps with self-heal — as you should — your emergency fix gets reverted by your own automation within seconds, mid-incident. The tool that keeps production honest is now fighting your remediation.
It doesn't propagate. The same fix usually belongs in staging and the next environment you build, and a hand-edit reaches neither.

The usual answer is process: restrict break-glass, carefully document how to turn off sync and back on, audit it, require a follow-up ticket.

The better answer is to remove the reason you reach for break-glass in the first place.

The fix, on the paved road

What the incident called for was straightforward: give the container an honest memory request, cap it with a limit, and set GOMEMLIMIT so the Go runtime backs off before the cap instead of slamming into it:

resources:
  requests:
    memory: 2Gi
  limits:
    memory: 4Gi
env:
- name: GOMEMLIMIT
  value: 3750MiB

In our ConfigHub setup, this goes into a unit called "deployment" (that's just our naming). There is one distinct copy of this unit for every deployment variant - dev, staging, production, etc. That's what makes it possible to do a pinpoint change, only to one specific environment like this:

cub run set-container-resources \
  --space prod-confighub-app --unit deployment --container-name confighub \
  --memory 2Gi --cpu "" --limit-factor 2 --operation all \
  --change-desc "Memory request 2Gi / limit 4Gi after node incident" \
  --protect
cub run set-env-var \
  --space prod-confighub-app --unit deployment --container-name confighub \
  --env-var GOMEMLIMIT --env-value 3750MiB \
  --change-desc "GOMEMLIMIT 3750MiB to match the 4Gi memory limit" \
  --protect
cub release publish prod-confighub-app

In this case we rely on the prefix naming pattern of the space (prod-<component>). But we could also have used a where clause to select based on labels and other properties.

The first two commands edit the authoritative configuration record — no YAML file to download, hand-edit and re-upload; each function writes exactly the fields it names, and nothing else: full audit trail, reversibility and team visibility. The last command releases this change as a new OCI bundle. Argo CD is immediately notified by argobot about the new release, pulls the OCI bundle and applies it. The new pod was running with the new limits in well under a minute, most of which was the pod restarting.

No CI pipeline, no template rebuild, no PR-merge-wait — and also no bastion and no kubectl. The path from a config edit to production is short enough that there is nothing for break-glass to short-circuit.

What makes this work

Three key properties of configuration-as-data enable this:

The path is short. An edit to a unit is live as fast as the delivery layer can reconcile it — seconds, not pipeline-minutes. Speed is not a luxury here. It is the difference between fixing the system while staying inside the config plane instead of throwing it away and accessing prod directly because of time pressure.

Low blast radius. Only the production deployment that had a problem was touched. If you run many production environments, you don't want to update config for the healthy ones in the middle of an incident. You first stop the bleeding and deal only with the problematic deployment with 100% guarantee that you are not affecting any other environment. Then later you come back and decide if this config change should apply more broadly. With configuration-as-data, each environment has its own literal config. So it's easy to touch only what you need to.

Anything is editable. In most config pipelines, there are a set of predetermined fields that expected to change via variable interpolation into templates, or other IaC mechanisms. When you need to stray off that path, it gets more complicated. You have to edit templates or actual source code to get what you need. What's the likelihood that the field you need to change to fix a production issue was anticipated in advance? Sadly not very high. Configuration as data does not have this problem. The unit is the resource: every field a Kubernetes Deployment has is a field you can change, without anyone having pre-blessed it. GOMEMLIMIT was not in anybody's golden-path template and it didn't need to be.

There's an additional feature that makes the fix safe and durable. The memory values now live on the production variant, downstream of a shared base that other environments inherit from. A later upgrade flowing down from that base could overwrite them. So we marked the three paths as protected by adding the --protect flag. We could also have done this in a follow-up command if we decided on it later:

cub unit set-protection deployment --space prod-confighub-app \
  --protect "apps/v1/Deployment:confighub/confighub:spec.template.spec.containers.?name=confighub.resources.limits.memory"

Protected paths are local overrides that upstream merges must not clobber. The emergency fix is now a first-class, recorded, revisioned part of the environment's configuration — not a snowflake waiting to be reverted, and not a sticky note saying "backport this later." Making staging match was the same edit against the staging variant, two commands, done. Later, we can decide, on our own schedule, if we want to upstream this configuration to a shared base. Once upstreamed, we simply remove the downstream protections and now we're back to scalable configuration management.

Speed with governance

None of this eliminates control and governance. A change description is included with every revision, the full history of the unit is queryable, and you can gate applies on approval when the change isn't an incident. You can turn the governance knobs up or down, depending on your general posture and the specific situation.

The role of AI

This whole incident and remediation was done interactively via an AI agent. A human didn't make a single edit to configuration directly. The agent diagnosed the problem. The agent made the changes in ConfigHub and, with human approval, rolled out the changes. Then the agent monitored production and verified that the fix was successful. The cub commands listed in this blog post were constructed by the agent itself, working entirely from the CLI's help text, the public docs, and the memory it has built up operating our ConfigHub deployment.

Here are some examples of the prompts that were used. First, once the problem was diagnosed, we needed to decide what values to set:

Please recommend a specific memory (and cpu) limit we should set on confighub to prevent this in the future.

The agent was already scanning Loki and VictoriaMetrics and knew the dimensions of the node pool. So it was in a good position to give a recommendation. Once the recommended limits had been reviewed, the next prompt was:

Ok, please go ahead and set this on the prod variant directly and enable field protection for the fields.

The agent made the change and released it upon human approval.

The agent never needed write access to live production config (we did need to kill some stuck pods but that doesn't mess with config). It only needed read access to production and write access to ConfigHub. It didn't even need release publishing privileges in ConfigHub. That can be reserved for the human approver without slowing anything down.

Configuration as data plus agent-led operations gives you speed, safety, and full governance with no reason left to break glass.

Try it yourself

ConfigHub Cloud Preview is free and open to anyone.

The tutorial walks the whole loop from this post — edit a unit, release, watch it land — on a local cluster in a few minutes. Then go edit something your current tooling never intended you to change.