“Configuration drift” usually refers to the drift between the source of truth for a configuration and the live state. But there can also be unintended or undesired drift between different variants of a configuration, such as deployments of an application in different clusters or environments. This is another flavor of the “works on my laptop” problem — works in dev, staging, some regions, etc.

How can this kind of drift be prevented? There’s an even more basic issue: how do people determine what the configuration should be for each deployment target? The answer to that is often parameter values.

I’ve written about over-parameterization of Helm templates many times, such as in the original document explaining the thinking behind kustomize. It happens both with off-the-shelf charts and with charts for custom applications.

Some people have fully embraced the “parameterize everything” approach. I call this the “One Chart” approach, after this Michael Goodness talk and Gimlet’s OneChart. The pattern is also used by the Helmet library chart.

Such generic charts often have template code that looks like this:

  {{- if .Values.foo.enabled }}
  foo: {{- omit .Values.foo "enabled" | toYaml | nindent 4}}
  {{- end }}

Or it might just check whether .Values.foo exists, just reference a single value, explicitly reference several keys and values, iterate over a dict, etc. But the pattern is essentially: if this key is in the values’ YAML, and, optionally, is enabled, then insert the corresponding value(s) into the template’s YAML here.

This approach doesn’t provide much or any abstraction over the raw Kubernetes resources, so what’s the point of doing this rather than just writing Kubernetes manifests directly?

  • Default values can be provided.

  • It formats the YAML for Kubernetes resources. The values.yaml structure is simpler — just nested maps.

  • Because values files are just maps of maps, they are easily mergeable, so values can be factored into multiple files and “stacked”.

  • In a few cases, the same value may be reused in multiple places, such as in both resource names and labels.

Compared to a simpler chart that exposes fewer parameters, it’s very flexible. It doesn’t hardcode decisions into the template and mitigates the need for patching via kustomize or other mechanism. Simpler charts can be created by wrapping a generic chart. Helmet, for example, was written with that intent. But even with umbrella charts generic enable values can be exposed.

Obviously, with the generic template approach, just by looking at the template one can’t infer what value(s) is likely to be inserted or, importantly, where (i.e., which clusters/environments) and why. Instead, the intent for what values should be set where is encoded in directories and files.

The values might be organized something like:

globals/
  global.yaml
  environments/
    dev.yaml
    staging.yaml
    prod.yaml
  regions/
    na.yaml
    eu.yaml
    ...
  clusters/
    staging1.yaml
    ...
    prod-naeast1.yaml
    ...
charts/
  cert-manager/
    base.yaml
    environments/
      prod.yaml
    clusters/
      prod-naeast1.yaml
...

There could also be values files used as more specific mixins, sometimes called traits (OAM, OpenChoreo) or aspects, such as:

traits/
  foo/
    no-foo.yaml
    default-foo.yaml
    special-foo.yaml

Each file represents a different configuration of that trait. These types of mixins can be used across multiple applications in the case of a unified generic chart like OneChart. Off-the-shelf charts, on the other hand, don’t generally share template libraries.

With multiple layers of values file overlays, one would likely build a tool to select and enumerate the correct values files in the correct order for each cluster, to implement the desired inheritance and override model. Like a poor man’s Hiera. Over time it requires a fair amount of discipline and oversight to eliminate special cases or move them up the hierarchy or factor them out into mixins. When that isn’t or can’t be done, then the same values may be set in multiple files, and may be different in others. That’s when variant drift becomes challenging to manage.

This kind of structure can address the what and where, but what about the why? If you’re lucky, there are comments in the values files. Failing that, you can do a git blame and track back to a commit message and/or PR discussion. If the value is set in multiple files, you may need to track down multiple commits, potentially across multiple git repositories.

After doing that, you may discover the reasons why the values are set the way they are, assuming the information does not conflict. And also assuming it matches the state in the clusters. Anyway, while YAML comments, commit messages, and PR discussions can provide context for AI agents, they are not usable by conventional tests and tools. When someone makes a change later, how can the correctness of the change be validated?

An intent-oriented mixin/trait approach might help some, but this approach doesn’t appear to be commonly implemented in public charts. The ArgoCD installation manifests are an example of splitting configuration by high-level intent: namespaced vs. cluster-scoped and HA vs. single-replica. A similar approach could be used with Helm values. Instead, combinations of values are documented.

Higher-level intent is not generally encoded explicitly in a machine-readable form: high availability vs. not, persistent vs. not, scalable vs. low cost, PCI-DSS compliant, etc. The values are set the way they are, spread across many values files factored by commonality, sometimes with up-to-date comments for humans and/or AI agents.

So generic Helm charts do not convey intent, and push the variant drift problem into values files. The approach transforms the configuration schemas from standard Kubernetes resources and well known CRDs to templates that are impossible to reason about locally and that have non-standard values schemas, which hopefully are specified with schema files, but have no widely understood semantics, at least outside a single organization and type of workload. This lack of standardization, combined with unidirectional configuration generation, makes configuration as code not interoperable with reusable tools: operational tools, security tools, access-control tools, and other tools.

We’ve been living with this tradeoff for so long that it’s easy to forget that it is a tradeoff. It’s time to consider alternatives not just to Helm, but to configuration as code in general. Kustomize showed that there are other ways to implement configuration inheritance and traits. ConfigHub builds on lessons learned from kustomize, kpt, and porch to not just provide an alternative approach, but an API for configuration that enables interoperable tools.

Intent attached to workloads in a well known form can be respected by tools, which are decoupled from the configuration data itself. In ConfigHub, expressions (called filters) that select what data operations (e.g., functions) should be applied to can be decoupled from the code, as well. I’ll write another post about how these mechanisms enable management of variant drift based on intent and properties of the configuration data. I’m excited about being able to develop a first-class solution to this problem.

Have you discovered undesirable variant drift? How do you organize your Helm values files? How many layers of values overlays do you have? Did you build a tool to generate the specification of which values files to use, or just use ArgoCD ApplicationSet templates with conditionals? Do you use non-trivial conditional logic in your templates rather than just generic enable checks? Do you have a way to indicate higher-level intent, such as by using values mixins, or by annotations on the workloads themselves? Do you have any way of validating that rendered values match higher-level intent?

Reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in my other posts.