Every team that runs more than a handful of Kubernetes clusters ends up inventing the same thing. You do not want a change landing everywhere at once, so it goes to one cluster first, then a few more, then the rest. Canary, then a ring (or ‘wave’), then the fleet.

The odd part is how much machinery that simple idea usually needs. Rollout groups, stage definitions, a pipeline with its own syntax, somewhere to write down which clusters are in which wave (or ‘ring’), and something to keep that list updated as clusters come and go.

We have spent a few weeks on this with Sveltos, and the result is very cool.

ConfigHub gives your Sveltos fleet an editable backend, enabling simpler, faster and safer changes. Change your cluster groups? Update the database. Filter and patch? Run a query. It is all declarative and GitOps friendly.

What is Sveltos?

If you have not come across it, Sveltos is an open source Kubernetes add-on controller. It runs on one management cluster and looks after the others. Our integration is on GitHub here.

You register all your clusters in Sveltos and give them labels, the same way you label anything else in Kubernetes:

  • environment=staging

  • region=eu

  • rollout=pilot

Then you write a ClusterProfile, which says which labels to match and what the matching clusters should run.

Sveltos works out which clusters the selector covers, installs the thing on each one, and keeps watching. If someone changes it on a cluster, Sveltos puts it back. (And yes, Claude can help with all this)

In its native design you never name the clusters. You describe them, and any cluster matching the description picks up everything that matches. That is a good design, it is why Sveltos scales to hundreds of clusters, and it is the first thing anyone likes about it.

Updates are straightforward. Sveltos leverages OCI and can use Flux or Argo too. In today’s post (and GitHub example) we’ve gone for a minimal set where ConfigHub publishes changes to an OCI gateway. Let me know if you are interested in other configurations!

Two models → One

Before the examples, it is worth saying how the two tools think, because they overlap in exactly one place and that creates a design problem.

Sveltos thinks in sets:

  • A management cluster

  • Workload clusters registered with labels

  • And a profile carrying a label query

  • The profile matches a set of clusters, installs the add-on on all of them, and reconciles them forever

  • Scope is decided at delivery time, by the controller

ConfigHub thinks in variants which are database records:

  • Configurations are records with revision history and approval gates

  • These records have a data model: variants, in a relationship graph

  • Variants are derived (revised) from a base variant

  • Revisions are declared (declarative) departures from the parent variant

  • Scope is a property of the record, decided at review time and stored

Therefore:

Both models can answer “which clusters get this change”. Without integration, you could end up expressing the same change set decision twice, in two places, with only one revision able to ‘answer questions’ afterwards.

The way we join them has three parts

ConfigHub owns what configuration is and whether a change may be approved. The content, the review, and the approval are bound to one exact revision.

OCI is the boundary from ConfigHub to Sveltos. An approved revision is published as an image on ConfigHub’s gateway. The handoff is a digest: immutable, addressable, verifiable from either side. Neither tool has to understand the other’s internals.

Sveltos owns delivery and live runtime convergence. It fetches that image, applies the reviewed profile to the cluster it names, and keeps it there.

The integration may be summarised as: one ConfigHub record per Sveltos cluster. Each record carries a selector that matches exactly one cluster, so the selector stops being a Sveltos fan-out mechanism and becomes a Sveltos address.

What does this give up? Matching many clusters with one profile is a Sveltos feature that this design does not need, because all cluster configs are explicit in ConfigHub. Sveltos takes care of fleet orchestration: it keeps every addressed cluster converged and repairs drift without anyone asking, continuously. Governance and authorisation derives from config changes.

Analogy: we deliver change across the fleet.

Think of a record office and a delivery service. The office issues the authorisation and says who it is for. The delivery service takes the sealed package to the address on it and makes sure it stays there. The only way it goes wrong is if the delivery service starts deciding who the recipients are.

One variant per cluster

You may wonder if this is efficient. The obvious objection is arithmetic. Fifty clusters, fifty records? Yes, and that is the point: fifty independently approvable, independently rollbackable, independently answerable records. Remember we want to be explicit about all changes because of AI.

What makes it cheap is that they are not fifty copies. They are variants.

  • One base variant holds what every cluster shares.

  • Each live deployed cluster corresponds to a deployment variant of that base and carries local changes: its name, the selector line that addresses it, and whatever that cluster genuinely does differently.

In our fleet example today, the pilot cluster’s config variant departs from the base variant in three fields, and one of them is the address. This is the whole declaration, from the committed example:

Promotion in a Fleet

Promotion is then the interesting part. You make a change once, on the base, and it flows outward to each cluster’s record, while each record keeps what makes it itself. Nothing is ‘clobbered’ or quietly overwritten by a careless upstream change in the platform, such as an expedited patch.

And a roll out “wave” is not fifty separate manual updates. It is one bulk operation over a set: select the records for this wave with a label query, upgrade that set in one command, approve that set in one command.

ConfigHub applies the bulk update query and records an approval per record, each bound to the exact revision that record held at the time. One operation, N recorded approvals, the mapping intact.

Bulk action and per-cluster truth are not in tension.

For fifty genuinely identical clusters where none of that matters to you, a delivery-time label query is the right tool and you should use Sveltos exactly as it is designed. The moment one cluster needs to be held back, or rolled back, or someone asks what production A specifically is running today, the mapping has to live somewhere you can query.

How they fit together

Let’s go to ConfigHub and look at a running Sveltos fleet.

how the configs are related. note that ‘base’ is not deployed

This fleet snapshot is from our initial set up in ConfigHub, showing the base variant on the left, and one deployed variant per cluster; then on the right, four workload clusters and the deployed management variant below. Each is on its second revision after one reviewed change to the base.

What about updating to live? This happens when we hit “release” in ConfigHub, and OCI images are published, like this:

OCI based updates

Here is the flow:

  1. Config comes from ConfigHub, where each cluster’s config variant is stored, checked, and held at an approval gate.

  2. A named person approves one exact revision, and ConfigHub publishes it as an OCI image on its gateway. Each variant publishes to its own OCI gateway address.

  3. Sveltos, on the management cluster, fetches that image and sends the reviewed profile to the one cluster it addresses.

  4. Sveltos then keeps that cluster aligned and repairs drift.

We have notes:

NOTE that we did this without an added GitOps controller and without an additional intermediate registry. Promotion does not change the wiring at all. Publishing the approved release moves the tag, and the addressed cluster follows, so approval alone moves the fleet.

NOTE: Reading a release from the gateway needs an add-on controller that decompresses gzipped layers, so every run used projectsveltos/addon-controller:v1.13.0-ch while that fix waits for a release. The same profile on stock v1.13.0 fails, because it tries to read the gzip bytes as YAML.

BOOTSTRAP NOTE: The management cluster’s own deployed config variant records the bootstrap profiles that tell Sveltos where to fetch from. Its first revision cannot arrive through the gateway, because it is what enables gateway fetching, so it is applied once at cluster setup, and ConfigHub governs every revision after that via the same approval gates.

The Integration between Sveltos and ConfigHub

records (variants) and update flows

Our GitHub integration features several use cases as “chapters”. They all have: one base variant, one deployed variant per cluster, one approval.

  1. Kyverno across the fleet installs admission policy through reviewed per-cluster variants, each behind an approval gate.

  2. Kyverno canary: using the same example as (1), the pilot cluster’s revision is approved and delivered first, and widening the rollout means approving the revision for the next cluster’s variant.

  3. Environment rollout promotes one reviewed values change from pilot to staging to production, wave by wave.

  4. CVE patching: one reviewed version bump with digest-bound provenance, closed by a coverage audit that proves no cluster was missed. No vulnerability scanning is claimed.

  5. Bulk operations: one reviewed edit promoted to every cluster’s variant in one pass, closed by a zero-drift audit.

Let’s take a look in stages.

Canaries made simple

We roll out the Kyverno open source software across the fleet.

Imagine two staging clusters, and two variants i.e. one per cluster, and both variants of the same base. Underneath is Kyverno 3.8.1, with three replicas of the admission controller.

YAML for this example

See below: the history of changes to the second cluster’s variant. We cloned from the base, armed its approval, then departed in exactly three fields including the clusterRefs entry naming its cluster, then approved.

kyverno roll out

NOTE: Admission policy is a good thing to be careful with! Since it decides what a cluster will and will not accept. So both variants sit behind an approval gate. ConfigHub held the pilot cluster’s variant revision, until someone approved that exact revision, published the now approved revision to the gateway, and Sveltos installed Kyverno on the pilot cluster. An unapproved record publishes nothing. So the second cluster got nothing: no release, no namespace, no trace. It is inert.

How to widen the rollout with one small config change

Widening the rollout is simple: now we approve the second cluster’s revision. There is no selector to edit and no list to maintain, and there is no moment where one approval quietly covers a set of machines. Each ConfigHub approval names one cluster and one revision.

Above: Read the revision activity from bottom to top: cloned from the base with the approval gate armed, departed in exactly three fields (its name, the line that addresses its cluster, its removal behaviour), then inheriting the reviewed change. The approval binds to this record’s exact revision.

Post approval it is then published with a different digest. And then Sveltos installs Kyverno on the second cluster while the first remains untouched.

Above: two deployed variants, shown after both waves. Each ships to a Target (~cluster name). Base variants are undeployed and have no Target.

Summary:

We did the whole rollout. Two clusters, two config variants defining them, then ConfigHub revision approvals, and (for release) we published two OCI digests. Then we set the admission controller to one replica on both clusters, and Sveltos put both back to three.

Why this is better than a pipeline

We think this is a simpler, faster and safer approach than many alternatives.

The config change approval is recorded upstream: as we revise the one variant that authorises the change for that one cluster. Not on a deployment somewhere downstream, and not on a ticket describing one. Nothing reached the second cluster until a named person approved the exact revision that let it in. Approving yesterday’s revision authorises nothing about today’s.

Rollback has the same shape as rollout: Every cluster’s variant has its own history, so you can put one cluster back on its previous revision and leave its twin where it is. Fleet tools that treat the fleet as one unit cannot say that.

And nobody had to build a rollout engine: Look at the diagram again: there is no component in the middle doing waves. Sveltos already delivers to addressed clusters and keeps them reconciled. ConfigHub already stores reviewed records with approval, history and variants. The rollout is what happens when you put those two things next to each other.

The same move, three sizes bigger

Once every cluster corresponds to a config variant, the other fleet jobs turn out to be the same move at different speeds. Our example has all three.

Environments

One reviewed values change, made once on the base: Kyverno’s background controller went from one replica to two. It flowed outward to four per-cluster records in three waves: pilot, then staging, then both production clusters. The production wave is worth looking at closely. One label query selected exactly the two production records, one command upgraded that set, one command approved it, and the receipt records two approvals, one per record, each bound to that record’s own revision.

Each record then published its own release, and prod-a and prod-b converged from different digests at their own gateway addresses. The digests differ by construction, not by luck: each record’s bytes differ at least in the line that addresses its cluster, so their releases can never be the same artifact. The per-cluster matrix shows which cluster ran which revision at each checkpoint, and the observed column comes from the run going and looking at the clusters.

Patch day

One reviewed version bump, Kyverno 3.8.1 to 3.8.2, with the patched chart’s provenance checked against the reviewed digest before anything was stored. It promotes outward through every cluster’s variant, the reviewed values from the previous chapter survive the bump on every cluster, and it closes with a coverage audit naming every cluster and confirming each one runs the patched version.

No vulnerability scanning is claimed anywhere: this is governed delivery of a version bump, not an advisory check.

Everywhere at once

One reviewed edit raised a replica count and promoted to every cluster’s variant in a single pass, closing with a zero-drift audit: no armed gates left, no record changed out of band, and drift injected on every cluster repaired.

ConfigHub publishes and never connects to your clusters, which is deliberate and is why it holds no credentials for them. So it does not tell you what is running.

Sveltos tells you, because it reconciles continuously and records what it did per cluster. This can then be reported to ConfigHub for additional operations.

Why ConfigHub

I said it was simpler, faster and safer. That comes from making all configuration explicit. Once we have done that, we have a central place to gather observed facts from observability tools and to control operations.

I highly recommend reading Brian’s post “There should be an app for that”. This tells you how easy it is to add more of your own functionality to the baseline fleet ops we have described. Just use the data in ConfigHub. If you can’t get the right data into ConfigHub, email me.

Some fleet apps you could build:

  • Cumulative failure budget that halts a rollout when failures cross a threshold

  • Automated verification step between stages

  • Timeout on a stalled wave

  • Red button halts and reverses across the whole fleet

None of those need a new engine underneath. They are queries and writes over data you already hold, which is why they are cheap to build here and expensive to build anywhere else.

ConfigHub is the data substrate:

We are the part that decides whether any of it is safe, if the rollout definition is a reviewed configuration, and where the approval binds to exact bytes.

Run it yourself

Every offline check runs with no account, no cluster and no network, and the repository has no npm dependencies:

git clone https://github.com/confighub/sveltos-confighub
cd sveltos-confighub
npm run verify

Node 22 or newer, python3 with pyyaml, and tar. The command runs the verifier and the self-test for every chapter, so a failure tells you which one.

The five-cluster fleet rehearsal builds its own clusters and needs no ConfigHub account to check it works:

HELM_EXPT_ALLOW_LIVE_SVELTOS_REHEARSAL=1 npm run sveltos-fleet-rehearsal:run

After the set up phase, the ‘governed’ chapters will need the cub CLI and one ConfigHub authenticated context, so that each runner checks its preconditions and, if it stops, does so with a named reason.

Conclusion

We believe the following is true if you are using ConfigHub.

Fleets do not need a rollout language.

They need a declarative data substrate and GitOps type fleet management of which Sveltos is a leading example.

A label query is used to select cluster variants in bulk at review time. Acting on the set enables a bulk approval applied per cluster, and individually recorded.

Widening a rollout means approving the next cluster variant’s revision, and that brings review, approval and history with it, because it is configuration like everything else.

Try it yourself: If you run a fleet, the repository is open, the checks run on your laptop, and I would like to hear from you if it falls over, please. You can easily run the example with a few kind clusters or in the cloud.

Finally, a companion example repository: kubara-confighub, tells the same story one cluster at a time, where the platform itself is the thing being governed. Blog post on Kubara is coming soon!

Onwards.

– alexis

get it done

Final note: At time of writing, ConfigHub cannot yet show live per-cluster status from Sveltos — the “what landed” flow back from the fleet — and the recordings used a patched addon controller (v1.13.0-ch) because the gateway serves gzipped layers and the fix is merged upstream but not yet released. When the Sveltos live-status reporter lands and when the fix ships in a Sveltos release, we will refresh this post. Both issues are tracked in the open GH repository.