GitOps in the age of AI — 4

Part 4 of a series. Parts 2 and 3 were theory. Now we apply that to a use case.

Set up the right stack, make sure it is running, check that it has the right add-on components, data and service integrations. Make a change. Patch the system. Replace the container image. Set up dev, stage and production. How are these related to your base system, to each other? What about fleets, how are your apps set up? Roll out a change in stages, maybe you have to check some tests and gates before promotion to production? Now do this “progressively”. You have lots of different use cases and your ordering of changes is dynamic, continuous. And now the boss team has arrived, bringing Mythos, and you have to patch all your Helm, yesterday!

This is when we can ask: do you have a configuration management system that can understand all this and automate it, using AI where possible, and provide dashboards for every job. In today’s post we shall look at a practical example of this, with generative GitOps: Promotions.

Promotions as Code or as Promotions Data?

Promotions, patches, image rollouts, config updates are versions of the same job: advance a change through your environments until it is running everywhere it should be. Mature GitOps teams already run these through one process: change the config in git, promote it with CI, add policy etc.

But look inside a CI promotion pipeline and, unless its well crafted to a simple case, you may find bespoke, imperative CODE. This is fragile. It can end in a broken state, half promoted, half not. It makes rollbacks difficult. Break glass? Another custom script? Every team writes this machinery, every team maintains it, and rarely is it part of the configuration data itself.

To manage this safely, CI owners will often simplify. They may take a snapshot and pass that ‘through’ the pipeline (or workflow). Maybe a githash names the state of a whole repository. It does not identify and describe the changes to declared config: which values moved, what they touch downstream, who may approve them, what proves they landed on each target. All of that lives in people’s heads and in the pipeline scripts.

Can we do better? Yes, if we can see changes as things, rippling out across a data set. This data set represents your operational desired state — your configs, your facts from the running systems, and the record of every change. A change can be a real thing, with an identity, moving through your environments. It can move through policy gates, leaving a record at each step, one operation each time. An operation on configuration DATA.

Traditional GitOps did not include promotions

A note on the GitOps state of the art for this, and what we hope to do differently with the goal of simpler promotions; and usage by other CD tools.

GitOps was never originally conceived to solve promotion by itself. The core reconciliation contract is target-local and does not specify how one release advances between targets.

I want to talk about why, and then talk about what can change today.

The argument goes like this: GitOps is a loop. One desired state, one environment, continuously reconciled. Then dev, stage and prod are three loops between three desired states and running environments. But a promotion is not a loop. It is an operation: the act of making and moving a change from one ‘dev’ loop, to the ‘stage’ loop, and then to the ‘prod’ loop.

The GitOps principles said nothing about it. Deal with it how you like! And this gave us flexibility: we could organise dev, stage and prod repositories how we wanted and not be tied down to some CICD/promotions vendor.

But people wanted promotions!

If you breathe, eat and sleep CICD then switch to GitOps, you will ask “give me back my pipelines!”. And the thing is they never left. GitOps moved the deploy step out and left promotions behind.

That created an ad hoc mix of declarative GitOps loops, and imperative job implementations. Not always with happy results: Promotion gets assembled from parts. Environment folders here. Cherry-picked commits there. CI jobs with approval steps holding it together. It adds up to a scrappy work-around for doing operations on configuration data, and is too unpredictable for adding AI safely.

Let’s “name the pain”: promotion pipelines in CI are imperative, fragile and brittle. They can easily end in broken states. They make rollbacks difficult. Can you break glass? It’s all bespoke, custom, imperative code. Maybe some humans can understand it and “get by”, but AI will barf.

Agents! Proposing changes by the dozen. What used to be contextual joins become a bottleneck. And user questions shift from “how do we promote?” to “how do we let an AI help without losing control?” And the answer is not “add more things, add AI config, make it more complex”.

We have to simplify.

How can we do this without creating a “Heath Robinson” assembly of repos, CI jobs, workflows, GUIs and magic buttons? We need a proper model for config operations, an extension to GitOps that can explain how config changes.

Prior Art and New Art

There is a ton of great prior art. Octopus, Harness, Kargo, and the vendors around Argo and Flux. Please see my footnote(*). We want to work with you.

A few things are new. We have a declarative model that treats configuration as data and adds relationships, schemas and expicit config management of variants in a graph; which also includes relevant context for dealing with generation of configuraton and transforms for infra and app operations. This lets us expand what GitOps means and make space for pipelines and promotions as data. In this post we write down an illustrative MVP for that.

We think this model is more suited to the machinations of AI tools and will enable combining deterministic control with non-deterministic AI outputs.

In the past operations have been seen mostly as some kind of step by step workflow. This mental model is everywhere: promotions, approvals, cost optimisation, incident management. WHY MUST IT BE THIS WAY? When using workflows, passing objects between processes, to get a job or “actions” done, then your implementations are either very simple or ephemeral; or will involve ever-widening transaction and event models. Could we instead, for some workloads, implement operations as updates on a shared database, ie. “apps”. With the simpler mental model and reliable failure modes of, say, Vercel or Lovable.

The model, in plain words

I shall describe a conceptual model: more ‘specification’, and less ‘implementation’ which is out of scope for this blog post.

A change (config revision) is a sealed (immutable) set of edits with an identity. Say, “image: 2.3 to 2.4”, sealed as change CS-42. Once sealed, it never alters. If the content changes, that is a new change(***footnote).

Your environments are variants of one base configuration: dev, stage, and prod, related and connected. In one example, prod is also three regional variants of one environment. We track what each one inherits and how extensions and override customisations work. This is a config graph and if you use Helm or Kustomize today then the graph is what your values files and overlays already imply, made explicit and queryable. My colleague Brian Grant has written about this in depth.

A promotion is one ‘governed’ revision: move this change to the next environment, but before it lands, gates check it; and after it lands, it is recorded.

Operational Context. Observed status, gate info, live checks and other ‘facts’ flow back from the running clusters. Is it reconciled? Is it healthy? Those facts, plus approvals, are what open the next gate.

One change, moving left to right. Evidence flows back to the ConfigHub store, right to left. Reconciliation both ways.

The pipeline itself is data. At its most distilled, it is an ordered list of environments and gates that the user declares: dev, then stage, then prod. And, each can be a set of targets: several prod regions, a canary group, a blue/green pair. The pipeline is not a CI system: it runs nothing, it only constrains what may move, to where and when.

This is a consequence of config as data in our model. Two properties matter most. The promotion machinery becomes declarative: an object you read and edit, not scripts you maintain. And it lives in the configuration store itself, as a first class citizen, next to the config it moves, instead of outside in CI.

Two more useful things follow.

1. First, it is additive. Declaring a pipeline changes nothing about your variants or their edges. It adds one small object beside them. You can change it, or remove it, without re-plumbing anything else.

2. Second, it is visible. You can view and edit it like any other configuration, in the GUI or from the CLI. Humans can read it. Agents can read it. So “where is my change” becomes a question the system answers, rather than a Slack thread.

Let’s look at three examples. Maybe these are jobs you already have.

Example one: an image rollout

First, the order itself. Someone on the platform team declared it, once:

Illustrative syntax of environments, an order, and checks at each gate.

That is the pipeline object, showing sequential promotion. So it is declarative i.e. the promotion sequence is stated but not executed as a job script. A simple interface but powerful capabilities exposed, where for illustrative reasons I have simplified and idealised a couple of things, eg. syntax, likely names, and showing small pieces of configuration, each written once, and edited when the process changes.

Notice what isn’t part of the pipeline:

The promotion sequence is not a property of the underlying config graph, which expresses logical config dependencies between base config, dev, stage and prod. It is not inferred from the config variants, or from a repo layout, or from labels and naming conventions.

It is declared: promotion order is a team’s process decision, not a property of the configuration’s structure. You change the process, you own the data, you edit the object.

And because the declaration went in as a governed GitOps change like any other “who decided prod needs an approval, and when” is itself on record.

Second question: where does the change come from? Nobody should be typing image tags into a box. CI builds checkout-service 2.4, and a registry event proposes the sealed changeset automatically, through the API. It lands in an inbox of available changes for this sequence:

_What this makes easy? Because an image reference is just a configuration field, image and config changes are the same kind of thing, so “this image with that config” needs no special bundling: you select which sealed changesets travel together, either alone, both, or neither, in one governed promotion. And unlike promoting a pinned snapshot, the combination is applied as changes against what each target currently runs: every field keeps its owner, the footprint is planned per target, and deliberate overrides can be protected inside the combined change. Want the older config back instead? A revert is just another changeset._

Now the rollout:

• checkout-service 2.4 is ready. The change: one field, image, 2.3 to 2.4

• in our simplified promotion sequence it is already live in dev

• you promote it to stage

• the system shows what will change before it does: one field, one target

• it applies, writes a receipt (= the record of what was done) and waits

• facts come back from the stage cluster: reconciled, running, checks green

• the stage half of the prod gate is satisfied; it waits for approval

• a human approves, and the change may move to prod

Prod is three regions, and this is a way for the config management graph to do some more work. Before anything ships, the fan-out is computed:

This map should come back before the change goes anywhere.

Now let’s review, or imagine an agent doing this: prod-eu keeps its deliberate override( → Note 1). No one had to remember the exception, because it is recorded as an edge in the graph and surfaced when the change is planned. Two records (‘receipts’) were noted, one per region that applied.

Then: when prod-ap’s health facts come back bad an hour later, that is not a “mystery outage to try to explain”, it is a recorded fact, it blocks the pipeline, and you can roll back ( → Note 2)

Notes

1. You can express the override in Helm today, a pinned value in prod-eu’s values file. What Helm cannot do is surface it at the moment a change is planned, attach an owner and a reason to it, or protect it from being silently flattened by the next edit to the layering. Context matters.

2. Standard tools should already tell you a rollout is unhealthy; that part is not new. What is new is that the signal is tied to this exact change and this attempt, it automatically blocks the rest of the sequence, and it lands in the same record as the change, so “what happened to CS-42 in prod-ap” has one answer (and you can roll back). Again: Context.

What this replaces: the Heath Robinson mess of the environment branches, the cherry-pick PRs, and the “did eu get the override reapplied?” thread, all of which come from tying imperative processes together with declarative GitOps loops.

In plain terms. We arein a model were configuration and context are stored as data, with relationships expressed as a graph. In this: the change itself has an identity and stays sealed; the promotion is planned against each target’s current configuration and not applied without checking first; also a value can have an owner that checked before the write; and the evidence (recorded context) of what happened is tied to the exact change and attempt, in the same record. Deployment products may capture some or all of this information.

We are doing it in one place as declarative GitOps in the ‘generative’ style.

Example two: Friday’s CVE, with an agent

In Part 3 we posed the 4pm on a Friday test. A critical CVE arrives in a base image, is patched and needs to be verified and deployed fleet-wide ASAP. Here is the machinery…

CI produces the patched image. This is not one service’s pipeline: the vulnerable base image is baked into many services across the estate. So an agent queries the desired state (config graph): which deployments run the vulnerable image, which configuration produces each one, who owns them. It proposes a single sealed change: replace the image digest, everywhere it applies.

Notice how this differs from example one. That was one service’s own release, moving through its own sequence. The trigger here is a security advisory, not a build event. So the first job is discovery, which is why the graph gets queried.

An agent asks: which deployments run images built on the vulnerable base, which configuration produces each one, who owns them. That is Part 3’s question one, answered as a query. CI produces the patched image, and the agent proposes a single sealed change: replace the digest everywhere it was found.

Could each service instead pick this up through its own normal sequence, like example one? For a single service, yes. What the fleet needs on a deadline is the campaign: one discovery, one approval, one rollout, one“are we done”. Fifty independent sequences could give you fifty local answers and no completion.

The system computes the footprint before anything moves. Every affected deployment is listed, along with the deliberate exclusions, each exclusion carrying its owner and its reason.

A human approves it once. Then the change rolls out through a pipeline whose stages are groups of targets, declared the same way as dev, stage and prod. Some deployment practices call these ‘rings’; others say waves. Note that in example one, prod was one pipeline element containing three parallel targets, in example two rings are several such stages in sequence.

The example below should illustrate the shape, though in a real system there would be more machinery both intra- and inter- cluster.

Each gate opens on detailed evidence that a machine can read. Not “the cluster looks fine”, but the exact new image digest, confirmed as reconciled and healthy in the previous ring. Then later if ring two stalls, the rollout holds. Note where the AI sits, because this is our generative GitOps safety plan: the agent proposed the change and started it moving, and it cannot certify its own success. The evidence is coming from the running clusters, independently, and we can store it in our graph of facts and config. Finally, when done: the proof that this work was completed is a query “what was affected, what changed, who approved it, where it landed, what was verified”.

What this replaces: the war-room spreadsheet, the per-cluster archaeology, “we think it’s everywhere.”. No more config hell!

Example three: two agents, one value

This is more of a ‘what if’ case: to see when AI is safe day to day.

A cost-tuning agent writes a change to the checkout service’s replica count (or maybe to the autoscaler parameters). The same afternoon, a second agent, working a performance ticket, writes to the same field. With writes sequential in time, does Git spot the concurrency? Not quite: with file/text based change operations in Git, whichever agent merges last will win cleanly, and the conflict info will be lost.

As set out in previous posts and above, in our model configuration is data and lives in a graph along with operational facts; so there are fields(**) with owners and we can do this:

Both writes were authorised at the resource level. The collision is named and held. The owner decides and full records are kept.

Git records what changed in the repository; it does not arbitrate. Two PRs to the same line are resolved by whoever merges last, CODEOWNERS gates files, and at dozens of agent changes a day nobody else is reading the PRs anyway. History should record that a contention existed and how/why it was decided. Named ownership of the value means the collision may also be arbitrated automatically, by policy, with both attempts on record.

This is a level of deployment control that 100% Git-based operations cannot easily express. We can add value by extending the data model (domain and store), eg. ownership of a single value makes dozens of agent changes a day a routine matter. (and no information is lost; as we set out in part 2 and part 3)

Summary: we did three jobs

We showed how the promotions pipeline and policy checks could be expressed declaratively as an illustration of the model.

1. An image rollout is a promotion through environments.

2. A patch is a promotion whose pipeline is rollout ‘rings’.

3. An everyday update is a promotion of one field.

We have been proposing a more comprehensive and modern version of GitOps that accounts for generation of declarative configuration ‘as data’, graphing logical relationships like ‘base config to stage config’, a data API enabling operations to transform this, and the expression of context for greater control eg. for analysis or remediation. The model is machine friendly: agents propose and initiate, and contextual facts are recorded.

By removing more of the imperative workflows people have been forced to use, we hope that CICD Promotions applications may be fully declarative.

This should lead to much higher certainty, lower error rates, and get platform delivery working “on Mythos time”. AI meets GitOps, via a shared toolchain.

No more Heath Robinson edge cases, no weird transaction breakages as workflow processes conflict at scale and under pressure of a high change volume.

How does this fit into my existing GitOps world?

We built ConfigHub assuming users who are already managing infra and apps on K8s with OCI, Helm, Argo, Flux, and other GitOps-type tools.

You keep Argo or Flux and use OCI for artefacts and updates. Your chosen GitOps tool then reconciles updates that are presented via an OCI endpoint, enabling their deployment into Kubernetes.

Your configuration lives in ConfigHub as variants. We help you manage their lifecycle and relationships via a data API that enables bulk ops and machine automation (AI). You declare a promotion pipeline (sequential flow) by listing your variant configs in promotion order. There may be policy checks declared here too. Changes are ‘sealed’ changesets. A new image version does not need typing in by hand: a registry or CI event can propose the sealed changeset automatically, through the API. Its promotion stays governed like any other.

Promotion: your agent presses the virtual button, the system computes the footprint, checks the gates, applies, and collects the evidence.

The view might be:

What is live right now and where a given change has got to. (TWO facts)

OK. One implementation question: what happens when a second change lands while the first is still mid-flight? The safe default is a queue: changes go through in order. You could say “just take the newest”, but this might silently drop the older changesets unless the newer one contains them. We think the system should surface this question through the UX (GUI, CLI, API) so that different users can answer it explicitly for their own cases.

What more might users want? Richer policy? Trigger automatic proposals to create enhanced changesets through our API? Automatic promotion after a soak period? Include a break-glass path, an “expedited” hotfix could skip the queue? All AI-led and ConfigHub-backed of course.

In our world, operations act on data (config, context).

This takes us into an “apps” model. That is to say, when a human needs to do promotions, they will want a promotions GUI for things like “showing all my pipelines in colour, with actions I must take”.

That is: a custom data-backed app on top of the core functionality on the graph. And that is much better than a complex transactional evented workflow!

Where this leaves us

None of this asks you to replace what already works. Git remains where your source code lives. Argo or Flux get simpler and apply the exact configuration that ConfigHub publishes. Desired and running state are kept in sync by GitOps; and your existing policies still run at the gates.

What changes is that promotion becomes an ordinary, governed operation on configuration data, rather than a process assembled from branches and CI jobs. Patches, rollouts and updates turn out to be the same operation with different pipelines.

And agents get exactly the role they are safe in. They propose changes and start them moving, under the same authority checks as anyone else. Whether a change succeeded is decided by evidence from the running systems, not by the agent that made it.

This is the foundation for generative GitOps: treat configuration as data; let humans and agents propose governed changes against the same graph; let independent observer tools return exact-version Ops Facts; and package each operational job as a focused ConfigHub App over shared governance.

Next in the series: Helm.

– alexis

Footnotes

(*) Why are we here talking about this when CICD products exist

We are proposing a wider declarative model than is easily accomplished using Git, CI, Helm and live Kubernetes. It is a configuration and context store with relations explicitly modelled so that Generative GitOps can capture more operational information and enable simple operational applications like deployment, promotion, cost estimation, remediation.

We would love this to be adopted by GitOps enthusiasts and anyone who is scaling their operational lifecycle using AI with Kubernetes.

Environment promotion is a longstanding Continuous Delivery discipline. Big platform products such as Harness and Octopus model environments, lifecycles, approvals, and deployment processes; GitOps-native tools like Kargo model artifact promotion around continuous reconciliation. Kargo is cool! It gives exact artifact bundles an identity (Freight), links Stages into pipelines, runs Promotions, verifies Freight, and visualises progress.

GitOps in its original form is about how one environment converges on declared state. Dev, stage, and prod can each have a reconciliation loop, but GitOps does not specify how a tested change becomes eligible for the next loop. Promotion lives between the loops.

The model set out here is narrower at the workflow layer and deeper at the configuration layer: it promotes a sealed semantic change over related variant configs (data objects). In contrast, Kargo and other CICD tools pass ‘freight’ between processes, which can create some operational challenges.

In summary:

• Octopus and Harness: broad deployment and release orchestration.

• Kargo: GitOps-native artifact discovery and promotion.

• ConfigHub: semantic, target-relative configuration transitions with context: authority, provenance, protected differences, materializations, and externally sourced evidence (‘facts’).

(**) This is not Kubernetes managed fields. Server-Side Apply does more than record ownership: a non-forced apply against another manager’s field is refused, real conflict detection, and credit where due. But it works per cluster, at apply time, and the “owner” is whichever manager last applied, not a declared policy. And in GitOps it is structurally blind to our case: both agents’ changes arrive through the same reconciler credential, one field manager, so no conflict is ever seen. Ownership here is declared in advance, checked at write time, before apply, fleet-wide, with collisions arbitrated and both attempts receipted.

(***) A commit names a snapshot of a tree; a changeset names a semantic delta over typed fields with a declared footprint. CS-42 is one identity across every target, and is applied as a planned delta per target, three-way, not a snapshot.