GitOps in the age of AI — 2

Part 2 of a series.

In my previous post I talked about how AI is changing GitOps. A particular problem arises when different ways to generate configuration get entangled with each other at ‘machine speed’. It is important to note that post-Mythos, now that frontier models do real operations work, all changes will need to be delivered both faster (<2hr SLAs) and in many ways more carefully.

I gave an example where all of this blows up — an innocent one-line change in Helm that is correctly processed by GitOps but still leads to a whole region falling over.

The proposed solution has two elements:

some of what the agent needed was knowable up front (structure, overrides, fan-out, deterministic, and thrown away by the generators like Helm/CI); and

some was not knowable up front, because it lives in the running cluster

So the deterministic part goes in a config store that remembers it; and the non-deterministic part arrives as context (operational facts and constraints). This post, part 2, is about what you build on that. Starting with the good news: GitOps doesn’t go away: it becomes Generative GitOps.

DISCLAIMER: Much of this post is aimed at users who want GitOps examples with AI. For everyone else it may seem too long. I apologise for this: and promise the next post will be for you. Above all please send feedback! It’s a team effort.

What happens when GitOps meets AI

GitOps gave us a reliable way to run systems: write the desired state down, store it in git, and let software agents pull it and reconcile the cluster toward it. The OpenGitOps principles captured that well: declarative, versioned and immutable, pulled automatically, continuously reconciled. And I think these principles gained acceptance by NOT specifying every little detail, so that implementers could interpret the model across more than just “today’s use case”.

The principles have stood up well. Yes there are gaps: promotions, fleets, and of course Helm. We’ll talk about all of these in this series of posts. And it’s true that the community has filled those gaps with solutions, patterns and products. But NOW we have a new use case: AI.

GitOps makes ‘silent assumptions’ about how configuration works: that a human authored it, that it lives as text in git, that people review it, that changes flow one way, from the config to the cluster, and that config is the only surface anyone edits while the live system simply follows.

AI breaks GitOps assumptions

AI is good at the source side of config generation: authoring charts, editing templates, manipulating files. For writing or reading a single component, a capable agent over text is fine, often better.

For this piece we focus on where GitOps intersects with AI: fleet configs, operations, searching, querying, bulk-updating and gating the rendered result over many environments at once. That runs over the generated output and its relationships — which are configuration as data.

Configuration is no longer mostly hand-authored. It is generated at machine speed. AI agents will generate it, indeed the current cohort are “generative” by design. This builds on prior programmatic generators such as templating tools, pipelines, and SDKs.

Unlike the GitOps reconciliation loop which ensures that data changes are kept, most config generation is forgetful. That means a program evaluates the relationship between an input and the output it produced, then discards it, leaving flat, explicit and literal YAML with no record of how a change to one value would ripple to others. Once such information is forgotten we lose certainty that a system change is correct; such obfuscation prevents GitOps from working. And the shapes do not match: generation is one to many, a single definition rendered across every environment, while reconciliation is one to one, each loop converging a single target toward a single desired state. Even a perfect reconciler cannot recover what the generator forgot, because the one-to-many structure was never in its inputs.

There is rarely just one generator, either. Helm, Kustomize, pipelines, and agents all emit config that overlaps, with nothing owning the seams between them, so more generators compound the problem rather than dividing it, unless every output lands in one graph that can see the overlap. And review does not survive contact with the new pace. When changes arrive faster than people can read them, and when the people are getting less expert because the agents are doing the work, the human in the loop stops being a safeguard and becomes a bottleneck.

And the live system stops being a downstream follower. Agents act across three surfaces at once: they change source code (and prompts, sometimes called ‘intent’, or ‘implicit’ config), they change desired state as represented by explicit configuration, and they change the running system directly, to fix an incident, patch a vulnerability, or trim a cost.

Traditional GitOps modeled only one of these, config flowing to the cluster, and treated a direct change to the live system as drift to be reverted. When agents write to the live system on purpose, that arrow has to run both ways, and every change, wherever it originates, has to be attributed and reconciled into one coherent picture. So the principles need an update.

I picked “Generative” because I like it and I don’t like “Agentic”. Isn’t everything Agentic now? I think that Configuration is Data. This provides a lot of the machinery we’ll need. But it leaves out the question of where config comes from. Much will be generated by one or more tools. We’ve shown how both deterministic and non-deterministic generators can get us in trouble. But they won’t go away. Who wants to read YAML? Even robots prefer JSON.

In future, I expect to see more configuration (and context) than we thought possible, and we’ll still need to track it, trace it and fix it.

Six principles for ‘Generative’ GitOps

The six below say what that foundation must add once AI is doing the generating and the changing. Three govern the deterministic half, the structure that a generator flattens away; three govern what happens at and after apply, where the live facts live. These principles should hold for all software agents: classic reconcilers, AI agents, and new hybrid forms. Yes I want to improve them with your help!

Generated Declarative Config is produced once, by an untrusted generator, and validated on entry; after that, the data is the source of truth, and nothing silently re-renders.

Held as data, not files. Desired state lives in a versioned, queryable graph with variants and relationships; everything Git guaranteed is kept.

Attributed. Every value records who changed it, when, under what authority, and why.

Proven and auditable. Every change carries machine-checkable proof, valid and within bounds, produced before apply, not reconstructed after an incident.

Bounded before applied. The blast radius of every change is computed and contained before it ships, along known edges with override protection.

Continuously reconciled, both ways. The classic loop survives and gains a governed return path: authorized live changes flow back into intent; runtime noise is observed, never folded back.

Say it again slowly

NOTE: The word “state” is widely used in Kubernetes: etcd has state, the live cluster has state. When I say the store holds the desired state, I mean declared properties that can be verified and correspond to what a human or agent says should be true, before any controller acts on it.

Generated declarative config. Produced by a generator, a template tool, a pipeline, or an AI agent, one time, as plain fully-rendered data (manual authoring of single files is a special case). The generator is untrusted, so its output is validated as it enters the system. After that the data is the source of truth, and every later change edits the data directly, increasingly by AI.

Re-generation? Ideally never; in reality a chart upgrade or a new platform default is a new one-time generation, and it must be controlled: deterministic, recorded, validated on entry, landed as a governed, diff-able change. Never re-render during release or apply, where output silently overwrites data with no record.

Held as data, not files. The desired state is versioned and lives in a queryable, API-addressable graph with first-class structure, relationships, and variants, not as text in a repository. A data API provides mutability but everything git provided is kept: every version retained, history complete, past states immutable. Change the substrate, not the guarantee.

Databases are good for configuration if we want to use AI combined with bulk ‘text ops’. Nobody drops Postgres or Cassandra for a folder of text files because search got good. Data lakes show that real data is heterogeneous. Configuration is no different once you have to query and change it across a fleet instead of reading one file. A range of config-as-data implementations are possible: keep source in Git, and add a versioned data store and OCI alongside it.

Attributed. Every value records who changed it, when, in which operation, under what authority, and with what stated intent, the way Kubernetes managed fields record the client behind each field. This is attribution of change, who altered the data, not provenance of derivation, where a value came from in the variant structure; that second relationship lives in the graph (e.g. for ‘bounded before applied’ below). Written on every change, by people, agents, and controllers alike.

Proven and auditable. A human reading every diff does not survive machine speed: the reviewer cannot read fast enough, and increasingly knows the system less well than the agent that wrote the change. So we strengthen the control rather than remove it. Every change carries machine-checkable proof, and the proof is the auditable artifact the control rests on.

By proof we mean: reproducible answers to four questions, recorded per change: is the config valid, did the change touch only what its author declared, was the author allowed, and where does it land. None of these questions is new. What is new is that at AI speed no human can answer them by reading, so the answers must be computed and kept.

Two checks produce them, before release (“apply”). The first check is validity: that the config is valid on its own terms, schema and policy. The second check answers the other three questions: is the change within known bounds; touching only the fields its author declared; where the author has authority over them, and can it reach the targets it claims. Correctness, then governance.

A human owns the policy, sets the bounds, and handles the exceptions; the evidence is produced automatically, before apply, instead of reconstructed by a person after an incident. I agree this could sound complicated, but prevention is better than cure, and it will get easier over time! (your help is welcomed)

Bounded before applied. Know your limits. The blast radius of every change is computed and contained before it touches a target, because it is rarely obvious: one input fans out to many objects, and a dense value touches many fields at once. Change flows along known edges, with override-protection, never by guesswork, and anything can be undone. The edges are the inheritance relationships the generator flattened away, recovered and held in the graph.

This limitation principle would have caught the outage in blog post part 1. The override and the fan-out were knowable before apply, and something or someone could have held them as data.

Continuously reconciled, both ways. Traditional GitOps was silent on the reverse direction; agentic operations need one. The classic loop survives, pull and reconcile, and gains a governed return path: when the live system changes on its own, an autoscaler, a hand fix, an agent’s remediation, the change is recorded with who made it, checked for authority, and if allowed it flows back into desired state through the same gates as any forward change. The filtering is the point: runtime noise, status, conditions, controller-owned fields, is observed and never folded back. The reverse direction is for intended changes to owned fields by authorized actors, and desired, applied, and live stay distinct.

When the live system changes on its own, an autoscaler, a hand fix, an agent’s remediation, that is not automatically an error to wipe out. It is recorded with who made it, checked for whether they were allowed to, and if so it flows back into the desired state through the same checks and gates as a forward change.

Net net: we added PROOF to traditional GitOps, generalised the store, and made the reconciliation flow both ways. This means we stop trusting only the people and code that produced the config, and start proving, attributing and bounding the config data and operational facts.

That is what lets generation run at machine speed without losing control. And it leaves Git doing what Git is genuinely good at, source, history, human authoring. What moves is the desired-state store, from files to data, in a graph, so it can be queried, attributed, bounded, and proven. You keep Git. You stop asking the filesystem to be a database.

From traditional GitOps to generative GitOps

You may recall the good old GitOps diagram. I think this was the first version of it:

Let’s update this picture (below). We have new labels and a bidirectional flow. I have moved some implementation details out of the picture (“into the background”) but there will still be: OCI gateways, source code in Git, configs as manifests, etc; but now the central control point is unified with the desired state store. I think this makes the new generative GitOps model superior to the traditional approach which combined Git+CI+policy, ad hoc rendering and hydration, distributed state at scale (in reality), and de-coupled operations management.

Example Revisited: our fleet change, six weeks later

We are in the same fleet as described in Part 1 of this blog series. Different agent, different day, same kind of job: “raise the CPU limit on the checkout service, we’re throttling.”

The agent doesn’t edit a chart. It writes the change to the config data directly and declares what it’s touching: one field, cpu, on the checkout service, everywhere it applies.

Before anything ships, the system answers back. The change fans out to fifty environments, and here is the map.

• Forty-eight inherit it cleanly.

• One, prod-eu, overrides this value on purpose, and the override is not a surprise buried in a values file: it is a known edge with an owner and a reason (“smaller nodes, keep at 750m”, platform team, March).

• With files, whether the change respects that override or wipes it depends on layering nobody checks. Here it is surfaced before anything ships, and the exception survives untouched.

• And one, prod-ap, would breach its namespace quota at the new value, a live fact read from the running cluster.

Traditional GitOps would discover this the way Part 1 did: at apply, per cluster, as a stalled rollout an hour later. But here the fact is read first, and the answer can come back before the change goes anywhere.

The same afternoon, a second agent, running a cost-tuning job, writes to the same field on the same service. Files have no answer to this at all beyond whoever merges last. Here the collision is named: both writers were authorized, the field has one owner, and the second write is held for the owner to resolve. Not solved by magic, solved by knowing it happened, which is the part files cannot do.

So the change goes ahead where it is safe: forty-eight environments, applied, each with a receipt. The exceptions come back as answers, not incidents: one deliberately excluded and why, one blocked and why, one collision held and why.

A person looks at three exceptions plus the collision already counted; yes: still three, fine, no change; just confirming I checked.

And finally: when prod-ap’s quota is later raised on the cluster by someone authorized to do it, that change flows back into intent instead of being reverted. That’s the traditional loop, but now running in the other direction.

Bidirectional GitOps.

Could a careful human have caught all this by hand?

• For one change, yes: go read the overlays, go check the quotas, go ask around.

• Post-Mythos, humans don’t have extra “reconnaissance time” for hundreds of agent changes per day. And no, you cannot script your way there by collecting dry-run diffs from every cluster: fifty local answers do not contain the relationship, which environments are versions of one thing, which overrode it on purpose, because that relationship was never in any single cluster’s view.

• At machine speed the choice is between review that reads everything, which is impossible, and review that reads only the exceptions, which requires a system that can compute the exceptions. That computation needs the edges, the owners, and the live facts held as data. Files do not hold them.

About that store

You’ll notice the store is doing a lot of work above: it holds the data, keeps the edges, records who did what, computes blast radius, and arbitrates the reverse flow. Fair question, what is it, who owns it, how does it back up, how does it resolve conflicts, how do you migrate onto it. The ConfigHub team has been writing about this and will continue to do so on our blog.

One answer worth giving now, though: the store publishes to Git and OCI, so Git remains your export, your audit copy, and your break-glass. A transport and a record, no longer the database.

Concurrency? When several authorized agents write the same value at once, who wins? We met this in the example above. Authority settles who may write but not what happens when two who do write collide. Last-writer-wins, leasing, ordering, field-level-ownership merge, are viable case by case. The choice will be a differentiator between implementations. Deliberately left open.

And Helm? We keep Helm like we keep Git. Of course we do. But in this story Helm gets deconstructed into literal config and custom constraints. This is what I see people doing in the industry already with emerging best practices: “Just use values”, “Compile CRDs into values”, “Don’t use hooks based on live, inside the chart”. More on Helm later in this series.

What this lets you build

A question I left hanging in Part 1: what kinds of apps can you write once configuration is data. Answer: operational dashboards for any jobs to be done. They are now DB-backed apps because config is data not files.

When config is files, every operational question is a bespoke script that reverse-engineers the fleet. When config is a queryable graph with structure, provenance and blast radius, those questions become small apps over a database. Show me every service over its memory quota. Show me which environments still run the image with the open CVE, and who owns them. Promote this, and prove it landed everywhere it should. On files, each of those is a crawler someone maintains and nobody trusts. On the graph, it’s a query, and increasingly an app an agent writes for you on the spot.

We’ve been experimenting with exactly this: small operational apps built against the config graph, a namespace manager, an RBAC manager, a network-policy manager, generated by AI and run against ConfigHub. Early days, but the point is already clear. Once operations run on data instead of files, “write me a tool that does X across the fleet” stops being a project and becomes a prompt. That means you can write ‘safe operational apps’ as one-shots.

Where to start

None of this requires abandoning your current tools. Render your config once, with whatever generator you already use, and hold the result as data: queryable, validated on entry, not re-rendered without checking first. That step works today and pays for itself immediately, because you can finally see and check what would actually deploy. From there, record who changes each value, recover the variant edges so changes carry their blast radius, and, when you’re ready, let authorized live changes flow back. Some of these ideas are laid out on our blog by Jesper.

The model: config is data, both-ways reconciliation, and agentic proof of change.

I said this was exciting. It is also really inspiring to work at ConfigHub with this team, on this technology, and then describe what we are building. These posts synthesise many elements from many people. This is the model we built ConfigHub on, and how we run it ourselves. If you’d like to see it working, we’re happy to show you.

Next up: more examples, more concrete pros and cons of all this.

– alexis

PS. While I was writing this, Taimoor Imtiaz shared this highly relevant example of a GitOps-AI wipeout and recovery on LinkedIn. Check it out! I love his assessment: “AI agents are incredible. They read code faster than any human. They find patterns we miss. They’re tireless. They’re confident. And they have no concept of what they don’t know.”