This is a follow-up-appendix to my last post.

A large organization runs at least 1,000 delivery teams. Each makes roughly 10 configuration changes a day. That’s 2.5 million changes a year.

Most land once. Some fail. A few escape to production.

What Happens When Configuration Fails

Consider a strong team: one deployment per day. Current DORA research shows elite teams deploy several times a day; and most teams deploy from daily to monthly. So once a day is a conservative number.

Each deployment carries several to tens of distinct configuration changes. Published pull request data shows the typical change is around ten files, of which configuration is a part. Call it ten logical configuration changes per team per day.

Now: How many go wrong and reach production?

DORA data puts the change failure rate, i.e. the share of deploys that need a fix or rollback, at roughly one in ten to one in five. A large share of those failures are configuration related. Peer-reviewed studies attribute between 16 and 47 percent of incidents to misconfiguration. Analysts estimate up to 70 percent of major outages. The Uptime Institute reports that four in five serious outages were judged preventable with better management, processes, and configuration.

Most detection is weak though. Industry benchmarks show that teams without heavy quality investment catch only half to two-thirds of defects before production. Configuration errors are harder still and often appear only in the target environment.

Put it together: a single team lets on the order of a few configuration faults reach production every year.

Across a thousand teams, that is several thousand configuration-caused production incidents a year. This is consistent with what operators report. As one customer put it, the ops desk already lives with tens of thousands of tickets, of which configuration is a large and rising slice.

The Blast Radius Problem

NB This is before we consider fan-out, which is what makes one mistake dangerous…

A component is almost never deployed once. It is rendered across many environments, clusters, and regions. So a single change re-renders across every target. Counted at the level of individual configuration values, the surface that can break runs into the tens of millions of field-level changes a day — and toward the billions across the largest estates.

This is a “blast radius”, not a count of human edits. It is precisely what file-based tooling cannot see or contain — just imagine looking at your YAML file and telling me what fields will cause the most damage if changed.

The Money Problem

The cost is not only the rare major outage. Start with the escalations: a meaningful share of those several thousand incidents become significant outages. The Uptime Institute (>1y ago) reported that most incidents now cost over $100,000. About one in five cost over a million.

But the larger cost is the ‘uncounted’ engineering hours to detect and trace each fault. The days lost to a single YAML error. The blocked and delayed rollouts. The standing toil of an ops desk already carrying tens of thousands of tickets.

Counted across the full population of incidents (ie. majority, and not just the rare long tail event) a large organisation carries on the order of a few hundred million dollars a year in configuration-caused cost. Minimum.

Roughly $300,000 per team. Before reputational or regulatory consequences. Before the wider drag on delivery, where one platform operator attributed 80 percent of rollout delays to configuration.

Now Add AI

And now add AI, which moves every term at once.

Recent telemetry across thousands of developers shows AI at least doubling the volume of change merged today. It heads higher as agents take over authorship. At the same time, DORA research finds AI adoption independently associated with lower delivery stability: a 25 percent rise in AI adoption tracked with a 7.2 percent fall in stability.

More change, produced faster, at a higher failure rate and with thinner human review. This compounds well beyond a simple doubling. Escaped configuration incidents rise four to five fold — toward 15,000 a year. The ops desk goes from ten thousand tickets to fifty thousand. The annual cost climbs several times over.

This is why the largest organizations now reckon with a billion dollars and more. As one customer told us: “We’ll go from 10,000 tickets to 50,000 in no time.”

The Question

No software process yet invented removes error. The question for a CISO or CIO who has to sign a compliance attestation is not whether configuration errors will occur. They will.

The question is: how many will reach production unseen? At what cost? And whether the current operating model can resolve them.

Today it cannot.

Questions or feedback? Email us at hello@confighub.com.

— alexis

References

  • DORA 2024, 2025: https://dora.dev/research/
  • Uptime Institute research on outage costs
  • Pull request data: typical change ~10 files
  • Incident attribution: 16–47% misconfiguration (peer-reviewed); 70% major outages (analyst estimate); 80% preventable (Uptime Institute)
  • Detection rates: industry benchmarks show 50–67% of defects caught before production