Four roads to config hell

When we started ConfigHub, we talked about getting organisations “out of config hell”. And we soon found people who wanted help.

You know the symptoms: outages, data loss, business downtime, dev and ops fighting to fix endless YAML and Helm. Then the diagnosis: we have lost control, costs are growing, performance is stuck, we can’t see what is going on, we are fighting misconfiguration and not delivering, do we need AI?

Our goal today is to give you four measurable “pain points” that you can easily recognise. These are causal factors that lead to bad outages, delays and losses. The idea is to help you quantify and prevent these from losing you time, data and money. We provide a free AI prompt to assist diagnosis.

Config Drift

Our first example is familiar: “configuration drift”. This is when “systems or cloud environments gradually move away from their intended, documented state”. That quote is from the security company Wiz.com who have a helpful deep dive page on the topic. Wiz go on to say that drift is to blame for “outages, security vulnerabilities, compliance failures, and unexpected costs” and caused by “manual updates, software patches, or environmental factors”.

There are tools that help you ‘fix’ drift, indeed one of the core principles of GitOps is for an agent to detect configuration drift between a runtime component and its desired state, and then converge or “reconcile” back to the correct state. But sometimes the problem isn’t so simple: you may have many such clusters and targets and what counts as “correct state” may be volatile or opaque.

Main config pain points

Drift isn’t the only way config can go wrong and bite you. Here is a table:

We shall explain these more below. But first note that everyone has these problems. Nobody sets out to build a bad configuration system. Careful people try to make good decisions. But for a bunch of reasons, most organisations can soon find themselves struggling.

One of the main reasons is that our tools don’t look after config properly. As we have argued in other blog posts, fundamentally configuration is data: and it is data about the state your systems are meant to be in, so if tools throw it away you lose the state, its provenance and history.

Imagine if a bank handled your money by using YAML files and CICD pipelines to execute transactions. Yet this is how we operate IT.

A warning: Take care with AI and configuration management. If you have lost transparency and control then AI-led changes could do more harm than good. “Why not just add AI?” can speed you towards further trouble. By the time those Helm errors are in production, it may be too late.

Config Sprawl

Our second pain point is “configuration sprawl”. TechTarget describe it as hidden software debt, growing through “fields, workflows, roles and settings that become harder to govern over time”.

Sprawl is what you get when configuration has spread across enough systems, files and formats that nobody can say with confidence which copy of a value is the one actually in force for a given live issue. Therefore, the cost turns up as lost time rather than as an outage, in the hours (or days!) spent looking for the right value and in the changes made against the wrong copy of it.

Consolidation is the standard approach to solving this. Maybe you put the shared parts in one chart, you layer environment values over the top, you keep a single source of truth. But as we have argued elsewhere, generating all your config from a template can lead to trouble at scale. Maybe the business adds a region or a team or a new class of cluster. The schema quietly becomes insufficient, then someone adds per-cluster overrides; but nobody looks after them.

Blast Radius

Third is “blast radius”, a term from safety engineering that infrastructure people borrowed. AWS makes an equivalent argument in their Well-Architected guidance on reducing the scope of impact and in their fault isolation material. The idea is to keep a failure limited to a fixed number of components rather than letting it spread “unbounded” or “too far”. A lot can go wrong when one small change is made. Just look at Helm charts with production based hooks.

The industry evolved canaries and approval gates to help. The concept: roll out in ‘waves’. And/or cap the number of files a single pull request may touch eg. “no more than five cluster files at a time”. But the rule does not distinguish between two very different operational cases:

• Change one line in one cluster’s own settings and you change one cluster.

• Change one line in a file that every cluster shares and you change five hundred

This is “reach”, and it comes from fanout: how many clusters pick that file up. The number exists, but it is not written down anywhere in the repository.

Example: This can really bite you: one engineer changes some cluster metadata. Another engineer changes a network policy. Neither looks like anything in review. Together they were enough to upgrade monitoring across the whole fleet, and nobody caught it because there was nothing in either pull request to catch.

What would have caught it is a number. How many clusters does this change? Put that next to the diff.

Config Limbo

Our fourth factor doesn’t have a ‘settled’ industry name, so we call it “configuration limbo”. A related phenomenon is ‘forgotten feature flags’ (eg here).

Limbo is: changes that were neither finished nor abandoned. Maybe a rollout that reached most of the fleet and stopped. Maybe a version bump went to staging, and never left. Most teams we talk to recognise this straight away. But they are hard to track, trace and prevent.

An example pain point:

Most charts depend on shared library charts, usually pinned to a range rather than a single version, with no lock file committed. That is often deliberate, because pinning exactly means a pull request per chart every time a shared function is fixed.

The pain shows up during a staged rollout. If a new library version is published halfway through, the early waves rendered against the old library version and the later waves against the new library version. Every cluster reports the same chart version. The manifests differ. Git does not record the wave split. OUCH.

Some tooling exists. Promotion systems, rollout controllers and release dashboards aim to sequence and track a change on its way through a ‘pipeline’. But what if a rollout that is halfway done looks like one that was deliberately stopped two quarters ago? Go and find the owners.

Measuring your Config

Point your favourite AI tool at a clone of your own repositories and ask it four questions.

Your AI will also want to know, if at all possible: what a loaded engineer hour costs you, how many patch cycles you run today against how many your security policy asks for, how long one target takes in a cycle, and how many of last year’s incidents started with a configuration change.

TRY IT NOW: There is a longer version, ten questions with the scripts an agent should write to answer them, in our public open source repo: examples/prompts

Measuring your Cost

Imagine an example. Take a fleet of five hundred clusters and a hundred and forty charts, twenty platform engineers at $110 an hour, monthly patching that you manage a quarter of the time, and six incidents last year that began with a configuration change.

Something over a million a year. Substitute your own numbers — and feel free to email me if you need help on how to structure estimates here and make use of the right tools.

Note how the two largest line items are not the cost of anything going wrong. They occur when nobody can work out what a change will touch. We put ‘top down’ numbers on that in a prior blog post: The Hidden Cost of Configuration Errors. More on cost analysis in future blogs.

Conclusion

As we said above this is about transparency and control.

Each pain point implies a question:

• What will this change touch

• What differs here, and why

• Where did this rollout get to

• What changed out there, and who decided it

Each question is about a relationship, between:

• A source config definition and the targets it reached

• A live value that differs, and the origin where that difference was agreed

• A decision and the edits that carried it out

• What was declared and what is actually running

You wouldn’t run a bank as a set of CI jobs on YAML and Helm. You’d have customer accounts in a database. When money moves between accounts, that’s a transaction on the database. That’s basically a ledger for customers, money and relationships between accounts. That’s what we need to get out of config hell — a single, central, transparent point of control. You keep your GitOps loop: Argo and Flux keep reconciling, developers keep authoring, and agents keep raising the rate of change; and all this needs policy and approvals.

Where to start

If you want to review your own private repos to see if you may be on the “road to config hell” then you can. Just start your favourite AI tool and give it a free prompt from our public open source repo: examples/prompts

Sign Up for ConfigHub and try the Tutorial.

Feel free to contact us directly.

— alexis

Note: cartoon is by Forrest Brazeal