GitOps in the age of AI — 3

Part 3 of a series.

Last week I posted about how we should update the traditional GitOps model to reflect new operational requirements from AI models and coding tools. Now we shall look at why this matters for applications and software operations. It is clear that for operational factors AI is an amplifier. AI is already amplifying existing risks which impact software operations. For example when IT is already overwhelmed, how can we safely increase the pace of change?

OBJECTIVE: we want to add AI and get measurably better at running software applications and operational “jobs to be done”. We are using GitOps and Kubernetes. What do we do?

GitOps: from tools to applications

In this post I want to look at our infrastructure and tooling for operating applications. We shall focus first on best practice tools. How well do today’s tools perform operationally, when “GitOps meets AI”? What needs to change? Is there a “stack”? What’s missing and how can that help even small scale users build the right applications which are safe for enterprise adoption.

Two quick considerations:

Rather than fine-tuning an individual tool, often the problem is how to think holistically about the architecture, observability..; all components, not just some.

Popular tools are often stretched past their original design goals, with diminishing returns. Then people ask for better tools. For example, Kubernetes did not originally have a packaging tool. Helm filled that gap and soon expanded to configuration generation, using source code, templates, CI tools and (sometimes) live clusters.

How GitOps loses information

As a friend said to me this week: “CI and Helm really suck for configuration”. AI can make this worse. In part 1 we showed how simple changes plus Helm cause information loss made worse by AI. To solve this we proposed an account of “Generative” GitOps, for which I set out six GitOps principles. The core ideas: config is data, both-ways reconciliation, and agentic proof of change.

A reader comment was: your outage example is not really an AI problem. A person who forgets the 768Mi override causes the same incident. AI may scale the problem, but it does not create it. YES. The problem is that we are losing information due to config generation and drift. For a decade, Helm, Kustomize, pipelines and CI have been throwing away the input-to-output relationships that we need to control complex systems. Long before agents arrived.

What AI does change is two things: it removes the compensating human, the person whose memory of the override and whose spare reconnaissance time papered over the gap, and it multiplies the volume of change flowing through the same information leaks.

Traditional GitOps loses information in two places.

1. At generation: the relationship between input and output, which base produced which variant, what fans out where, is computed and then discarded.

2. At drift: someone or something changed the live system, with an identity and a reason, and the revert erases both without a record.

Why do these “info leaks” matter? Because lost information cannot be checked. Once the fan-out map is discarded, no test can ask “what does this change touch.” Once the drift’s author is erased, no test can ask “was that allowed.” The checks do not disappear, they slide to after the fact, where they are called incidents.

Ten tools, and why they matter

So which tools should we look at? The ones fleet teams actually run. Almost every tool below exists as a response to lost information due to some leakage. A pattern can exist because the information leak hurt someone, even if it never hurt you :-)

1. Rendered manifests, hydrators, Gitless GitOps. Often promoted as current best practice. The GitOps industry converged on these after enough teams learned that the manifests reaching the cluster were not always the inputs that had been reviewed. Render once, freeze, sign, apply exactly that.

2. CI and pipelines. Simply where much of the delivery automation pipeline lives; nobody chose CI for configuration, rendering just moved in.

3. Helm diff. Obviously: see what a release would change before you apply it. You ran it the first time someone showed you.

4. Argo and Flux previews. They came with the reconciler: server-side diffs and dry-runs, a prediction per cluster before anything lands.

5. OPA, Kyverno, admission policies. Compliance asked, platform teams answered, and policy-as-code became table stakes.

6. Kubernetes RBAC and Server-Side Apply. The platform ships them; you configure rather than adopt. SSA’s managed fields are an underrated field-level ownership map.

7. Argo Rollouts and canary deployments. Progressive delivery is now received wisdom for several types of team…

8. ResourceQuota dry runs. Ships with the platform and is usually switched on by the team that got paged.

9. Secrets tooling: Sealed Secrets, SOPS, External Secrets. Git config is readable so act accordingly if you don’t want it to be read.

10. Sveltos. For me one of the exciting new tools. Deliberate adoption by teams already managing fleets, the crowd that already thinks in one-to-many (see this 15,000 cluster example!).

Plus one entry at the end that is not a tool at all: drift. We will finish there.

STOP! “Just give me the punchline.” If you want to skip the ten-tools deep dive then please jump down to the graph data and applications section.

The tool analysis framework

Which principle does it already implement? Of the six Generative GitOps principles, most teams are already running partial implementations without calling them that. You are further up this ladder than you think.

Where do its checks run? This is the shift-left diagnostic, and it is how the leaks show up in practice. For each tool, ask when and where its checks execute: before apply or after, one target or the fleet, and what information the check would need that the substrate already threw away. A right check in the wrong place, or at the wrong time, is a leak, showing.

What if we have an emergency at 4pm on a Friday? A critical CVE lands in a base image used across your estate; the CISO wants it patched fleet-wide, and proven patched, ASAP. Three questions define the weekend:

which of our hundreds of deployments actually run this image, specifically which config, owned by whom — not just an image in a CVE repo

what breaks when we bump it;

how do we prove on Mythos time* that it reached everywhere.

These questions are asking what the leaks destroyed: the first two need the fan-out map (leak one), the third needs a record of who changed what (leak two).

(*) Not by Monday any more! the same Mythos class of models that will help you patch is already helping someone else find what to exploit, so the gap between disclosure and weaponization is collapsing from weeks toward hours.

The tools, one by one

Rendered manifests, hydrators, and of course: Gitless GitOps. The furthest the ecosystem has already gone, and if you run this pattern, keep running it.

Implements: the generative principles that rendering should be done once.

Where the checks run: at publish time, which is the right time, on the wrong object. Signing verifies the blob: what arrives is byte-for-byte what CI produced. It cannot check what changed inside, whose values moved, what they fan out to, whether the author was allowed, because the render that knew those things has already run and discarded them, and an OCI artifact can always be pulled and inspected, one blob at a time. What it cannot then answer is semantic questions: whose values moved, what they fan out to, where this artifact landed, without unpacking everything everywhere.

4pm on Friday: the signatures help with question three, but the answers live in branches and blobs, so question one is still a script.

Note about Gitless: this splits the truth in two, humans authoring in git while controllers trust the registry, connected by a CI tool whose write discipline keeps them causally consistent. A risk factor is eg. two agents operating on resources with hidden overlaps between names, spaces and scopes. (The registry itself still earns its keep as a transport gate for Argo and Flux; the implementations are widespread and high quality.)

CI and pipelines. The quote at the top of this post is about this.

Implements: the mechanics of Generated, run on a schedule.

Where the checks run: in the pipeline, before apply, which is the right time, but a pipeline checks its inputs and then vanishes. CI is a workflow, not a store: no memory between runs, no edges, no picture of the fleet. It produces the rendered result and forgets it, which is leak number one, run as a managed service. Any check that needs last week’s render, or the neighbouring environment’s, has nowhere to stand.

On Friday: CI can rebuild anything and answer nothing; all three questions come back as scripts someone has to write.

Helm diff.

Implements: a slice of the bounded principle, one target at a time.

Where the checks run: before apply, which is correct, but limited to what one release can view. Diff compares a release against a target. Which of fifty environments inherit the value you are changing, and which overrode it on purpose, is not visible from inside any single release, because that relationship was discarded at render.

4pm on Friday: question two is answered for the release in front of you; questions one and three have no answer.

Argo and Flux previews. Both major reconcilers already compute a result before the apply step: Argo’s server-side diff sends a dry-run server-side apply to the Kubernetes API and compares the prediction against live state; Flux does dry-run and diff through the kustomize-controller and flux diff.

Implements: the Continuously reconciled, both ways principle, plus the ‘local preview half’ of Proven.

Where the checks run: before apply, per cluster, exactly where a reconciler should check. The local preview answers “what changes on this cluster, against this cluster’s live state,” and not “which of these forty environments inherit this value, and which have overridden it.” That second question needs a cross-environment data structure that understands variants and relationships.

We could try “just run the dry-run across all the clusters and collect the results,” but even joined up, this cannot tell a deliberate override from an accidental divergence, this cannot tell you which environments should have changed and did not, and cannot gate a promotion against the relationship, because the relationship that “these forty are instances of one thing” was never in the per-cluster view to begin with. A dry-run tells you what would change on a box. A graph of variant configurations tells you what is meant to be the same across boxes, and therefore what a change to the shared value actually means.

On Friday: question two, answered per cluster; questions one and three not at all.

OPA, Kyverno, and admission policies.

Implements: the validity half of the Proven principle, carried over unchanged: schema and policy. Making good use of your platform team’s rules.

Where the checks run: before a change takes effect, in CI or at admission. This is good, but limited to local not fleet concerns. A policy engine evaluates the document you hand it, and the facts the governance half needs are not in the document. These might include the author’s declared scope, the ownership of each field, the fan-out of the change. Also therefore, it cannot answer across the fleet and be sure that a change stayed inside its declared blast radius, and cannot confirm the author had authority. To check one change against rules is not to query the fleet estate. (Exercise for the reader: try writing a crawler to report on policy changes across a non-homogeneous fleet. Not a simple task.)

On Friday: it guards each change on the way in, and can ask the fleet nothing, so questions one and three go unanswered.

Kubernetes RBAC and Server-Side Apply.

Implements: the Attributed principle, in a limited style where managed fields record which client touched which field.

Where the checks run: at admission, the right place, with the wrong identity, and the ledger is written after ‘apply’, per cluster. The credential presenting a change belongs to the GitOps tool, whether it inherits Kubernetes RBAC like Flux or runs its own like Argo, so the actual author is invisible to the cluster and the managed-fields record mostly names the reconciler. And RBAC is all-or-nothing per resource: anyone who can update a Deployment can change any field in it. Deciding who may change one particular value has to happen where the desired state lives, outside the cluster. (Subresources like /status can help, but are tricky to use.) Right primitive, wrong place and time: the model relocates that record to the store and gives it what the cluster never keeps: history; this is written before apply, across the fleet, so it can be checked rather than merely read after an incident.

4pm on Friday: 4pm on Friday: attribution exists, but split across three systems, git knows an author, Argo knows who synced, the cluster knows the reconciler, and none of them is joined to the field that changed. Question three’s audit is a join that nobody owns.

Argo Rollouts and canary deployments.

Implements: the Bounded principle, in time.

Where the checks run: during the rollout, after apply has begun, watching one target. That is the right design for its job: bound the blast radius in time, roll out gradually, watch, stop. What has no check anywhere is the other dimension: which of fifty environments does this change touch at all, and which are exempt. A canary bounds blast radius in time; the graph bounds it in space. Complementary axes, not competitors, and Bounded needs both. Keep your rollout strategy; the graph tells it where it is allowed to run.

On Friday: each cluster’s image bump gets safer, and nothing says which clusters, or proves the set is complete.

ResourceQuota dry runs.

Implements: an ops fact, read by hand.

Where the checks run: at admission, per namespace, the latest possible moment. The signal is real, the prod-ap catch from Part 2’s example is exactly this signal doing its job, but read at apply it is a stalled rollout someone notices an hour later. Read before apply, as a fact in the graph, it is an answer that comes back with the change plan. Same check, moved left and widened.

On Friday: question two, one namespace at a time, and only if someone remembers to run it.

Secrets tooling.

Implements: Held as data, for the one field class that must never render: the reference is typed data, the value deliberately late-bound.

Where the checks run: first, note these are two different models. Sealed Secrets and SOPS put the encrypted value itself in git; External Secrets keeps only a reference in git and resolves the value inside the cluster. Either way, resolution happens at the cluster, which for secrets is correct by design: the value must not shift left, ever. And inside one cluster, the obvious question is answerable: “which workloads reference this secret” is a kubectl query away.

The missing check is the fleet half, and it has nowhere to run: across the whole estate, which workloads reference this key, what breaks when I rotate it, and did every one of them actually pick up the new version. In the model, the value still never enters the store; the reference does, as typed data with an owner and an expected version by digest, so rotation becomes a ‘governed’ operation; then who references the key is a query across the fleet, the fan-out is bounded before it ships, and receipts prove the new version was picked up everywhere, by digest, without the value ever crossing the store. Proof without disclosure ;-)

On Friday: rotation is the Friday test with a key instead of an image, and today it is the same weekend.

Sveltos. Teams adopt it because they already think in fleets. Profiles select clusters by label and deploy add-ons in a fan-out style to cluster resources that match.

Implements: the fan-out half of the graph, as cluster selection.

Checks: a profile matching a label set is a one-to-many relationship made explicit. That is a thing most of this toolbox lacks. Selection is resolved before deployment: the right time. Sveltos then goes further than most here: DryRun mode produces ClusterReports, a genuine per-cluster preview of what a profile change would do before it does it, and ContinuousWithDriftDetection watches what it deployed and reconciles it back. Preview before, watch after: two checks in the right places. Then, the reports are about per cluster and per profile, not per change. There is no receipt tying an individual change to its author and its provenance, no view of who moved which value inside the profile’s payload, and drift detection restores the deployed state rather than asking whether the live change was authorized, so there is no governed reverse path that folds a legitimate live fix back into desired state. Sveltos ‘knows fan-out’ better than anything else on this list; what it does not carry is the per-change story travelling along it.

We need a data substrate to retain information across tools

Could we build a ‘best of breed stack’ to implement the six updated principles? Almost. The tools have information gaps (as described above) and are missing some kind of substrate to glue it all into one consistent system. We can see this if we map back to the six principles.

• Generated. We can render-once in CI but the render lands as a commit, not as addressable data validated on entry. The next pipeline will re-render over it.

• Held as Data. A repo of rendered YAML manifest is ‘held as data’ in the same way that a folder of CSVs is a database: all of the content, none of the semantics. No edges, no variants, no write path with checks. You can read it. You cannot operate on it.

• Attributed. We could use git blame and managed fields. When the committer of record is the CI bot, blame answers “which pipeline ran” not “who.” Managed fields record a field-level manager (per cluster after apply) and the manager is usually the reconciler, because the tool’s credential masks the author. We end up with post-hoc and per-cluster information, but we need pre-apply and fleet-wide.

• Proven. Use OPA/Kyverno? This provides validity, schema and policy. Existing rules may be reused. The gap is fleet governance. A policy evaluates documents, for wider checks the required inputs are relationships that are not in that document: the author’s declared scope, the ownership of each field, the fan-out of the change. We also need transitions: was this change allowed, by this author, within these bounds. You cannot write that rule in Rego: the inputs live in the resource graph, not in the resources.

• Bounded. A helm diff shows the change on a target you pointed it at. Bounded is the other way round: it tells you which targets to point at, and which will refuse on purpose, and that needs the config relationships, which no diff holds.

• Continuously reconciled, both ways The reverse loop can be performed by hand, with no gate. It works until two agents do it at once ‘before the PR merges’. Doing both safely needs authority on the live write, ordering against the revert, and a receipt either way.

AI Applications operate on live systems

A data substrate could capture and retain operational information. This especially matters if we have AI agents operating directly on live systems.

Think of the classic “Kubernetes dashboard”. According to the very early versions of GitOps, you should not use this to make direct updates to the system. Even if updates are needed to fix operational problems quickly, they may conflict with the single source of truth (desired state) represented in Git.

Well, OK. But: an autoscaler, a controller default, and an agent’s security fix are all “the live system changed itself,”. Traditional GitOps gives you a menu for drift: revert it (selfHeal), flag it and wait (OutOfSync), or ignore the field entirely (ignoreDifferences, field management).

We are going to need this, and more, if we want AI agents to help us operate safely and at scale. We just need to be careful: Revert erases the context; ignore waves it through unattributed; flag leaves a human to reconstruct it. When all drift is wiped, who keeps the information? That is ‘leak number two’, the mirror of the first: where generators like Helm can wipe critical config related info, the reconciler can erase facts on the way back.

The missing contribution here is not detecting drift, it is refusing to destroy what it tells you: attribute it, resolve it by authority, and either fold it back or revert it with a receipt. When the live system changes on its own, do we know who or what changed it, whether they were allowed to, and can we decide per case whether it flows back or is reverted.

In our Generative GitOps model, live updates count as operational facts, and may record who changed a value or why. AI agents will be making decisions about live state ‘facts’ and acting on them. Already we see this with “FinOps” tools in AI platforms. None of this is solved without a suitable operational store that tracks these changes.

This is another area where Git alone (or Git plus OCI) is not a very good fit. Note what is new there: the AI agent is the first actor that writes to all three surfaces, source, desired state, and the live system, at machine speed, which stretches the traditional GitOps loop.

Applications on an Operational Data Store

In the new model, we are using AI agents to modify the whole system, aka Jobs to be Done:

Preview a change

Gate a release

Check a policy

Verify a rollout

Rotate a secret

Update the dashboard

These are applications that manage config and operational facts on a database (‘the graph’). They are no longer complex “workflows” with transactions, state, events and ordering.

Before, GitOps was how we managed K8s clusters and workloads. Applications were something else we could deploy after clusters and services, on top, maybe with Argo UI. In the future there will be one, simplified model for the whole stack: apps, services and infra.

Run the 4pm Friday weekend story again, but imagine we have a data graph of config and facts.

Which deployments run the image? Your scanner has an inventory and asset databases and cluster queries will list where the image runs. What none of them can tell you is the other half: which configuration, owned by whom, produces each of those deployments, and through what path you would change it. On the graph that already exists, the image-to-deployment- to-config-to-owner chain is data, so the answer comes back ready to act on, not just to read.

What breaks when we bump it? Blast radius computed along the graph edges before apply, deliberate overrides surfaced with their owners, quota facts read from the fleet.

Prove it landed? Receipts, one per target, aggregated into the report the CISO actually wants.

All completed in Mythos time, and signed off by the CISO before we go home on Friday.

Because all we needed was an app. The fleet-change job, written as a short program over the graph. HOLD that thought. In the past, each job grew its own tool, its own product category even, because files gave it nothing to build on, so every job rebuilt its own storage, its own fleet awareness, its own audit trail, badly or not at all. Its own dashboard. Put the operational desired state graph underneath and each job stops being a product category and becomes an app: a small program over shared primitives, the edges, the gates, the facts, the receipts. That is what compose means here, concretely. The same Jobs To Be Done are now an app catalog.

Keep your rendered manifests, managed fields and validity rules. Keep Git, keep Argo and Flux, keep the policies and the rollout strategies. What you add is the graph store underneath them, and the store is what lets them finally compose so the jobs become apps over the store.

For decades the missing IT component was a human. The engineer who remembered which region overrides memory. The one who knew why that field drifted last March. The one with a spare afternoon to go and check. The information the tools threw away survived in people. That is what “it worked fine” meant: a person was quietly plugging the gaps. Agents end that. The fix is not: a better human. The information has to move from heads into the operational graph.

The model: config is data, both-ways reconciliation, and agentic proof of change.

– alexis