“Configuration as Code”, the de facto best practice for managing configuration for the past two to three decades, has two defining characteristics, which are usually conflated:

  1. Configuration is represented as code that generates the actual configuration, and input values to the generator. Helm templates are the canonical example for Kubernetes, but the category also includes configuration languages (CUE, Pkl, KCL, Jsonnet, HCL, Dhall, etc.) and general-purpose languages (cdk8s, Pulumi).

  2. Configuration is stored and managed like code: kept in version control, diffed and code-reviewed, and deployed through CI/CD or GitOps.

Each idea rests on assumptions worth examining, but the second is largely a consequence of the first.

Our conclusion, after years of working on the problem, is that configuration should live in a database with an API — not in git. My cofounder, Jesper, recently wrote a post on this topic. This is my perspective.

Configuration represented as code that generates configuration

Start with the most common case in Kubernetes. You want to roll out a new release by updating a container image. With Helm, you template the field:

containers:
- image: {{ .Values.deployment.main.image }}

and supply the value through values.yaml.

A larger (but still small) snippet:

{{- $healthchecksPort := (default (.Values.ports.traefik).port .Values.deployment.healthchecksPort) }}
{{- $healthchecksHost := (default (.Values.ports.traefik).hostIP .Values.deployment.healthchecksHost) }}
{{- $healthchecksScheme := (default "HTTP" .Values.deployment.healthchecksScheme) }}
{{- $readinessPath := (default "/ping" .Values.deployment.readinessPath) }}
{{- $livenessPath := (default "/ping" .Values.deployment.livenessPath) }}
readinessProbe:
  httpGet:
    {{- with $healthchecksHost }}
    host: {{ . }}
    {{- end }}
    path: {{ $readinessPath }}
    port: {{ $healthchecksPort }}
    scheme: {{ $healthchecksScheme }}
  {{- toYaml .Values.readinessProbe | nindent 10 }}

Helm’s template syntax does create some specific challenges, but the structure would be similar in any language.

It’s important to recognize that adopting this approach is a tradeoff.

It has benefits, which is what led to the adoption of the approach, notably that it provides a recipe for generating multiple variants.

In this case, one would provide multiple different image values. These could be variants of the same workload in different environments, or a shared package that could be reused by different users for different use cases.

Some of the benefits attributed to configuration as code are actually benefits of any systematic serialization of the desired state, including fully rendered, WET (Write Every Time) configuration. These include:

  • pre-deployment validation and policy enforcement, though the configuration would need to be generated first if using an as-code representation

  • automated repair of the live state from the desired state / disaster recovery — a backup of the infrastructure, basically (though not of application data)

  • comments can be added to explain why values are set the way they are (except for JSON)

However, it has significant limitations and drawbacks.

As I recently explained, a huge opportunity cost is that building reusable tooling for changing complex code formats containing computation such as variables, conditionals, and loops is largely infeasible outside of what IDEs and AI can do. It requires compiler-based techniques, unless tools are just constrained to changing input values.

Instead, imagine dashboards that enable you to view and change Postgres versions across your fleet, to find and fix insecure Kubernetes pod configurations, change autoscaler settings, or to migrate off of deprecated API versions. We want to build a System of Record where changes could be made to authoritative configuration records across an organization.

Configuration as code is also challenging to use effectively, especially across an organization with developers, security staff, and other roles who need to use systems managed via configuration as code who may not be expert users of those tools and/or systems.

  • Unlike when using GUIs, TUIs, CLIs, and other user interface surfaces built around specific APIs, the template/generator author needs to understand the API details passed through by generic configuration tools: resource types and versions, field names, etc. This is expert knowledge, generally the domain of platform and DevOps engineers. In many organizations, this expertise is a bottleneck. AI can help now, but still requires humans to sanity-check the result. Reviewing changes made by AI agents can be harder than making the changes manually.

  • Additionally, there is no step-by-step guidance, progressive disclosure, out-of-the-box multi-step workflows, contextual help, error resolution assistance, navigational aids, dynamic updates, graphical representations, or other UX affordances. It’s just text. The developer-centric representation is especially hostile to non-developer stakeholders.

  • Moreover, there’s the 200% knowledge problem: the template/generator author must understand the syntax and semantics of the configuration, as described above, and also the machinery that generates it. And it’s not just the configuration author that needs to understand it. Anyone operating a workload in production and/or responsible for the scale, reliability, cost, and troubleshooting will need to understand the configuration that is applied to the system under management, Kubernetes in this case, and to make changes would need to understand the generation machinery to some degree.

  • These generators are unidirectional. To make any changes, one must change the generator code/templates and/or the input values, then regenerate. The tools assume that all changes will be made through the tools, which is known as exclusive actuation. This “factory model” adds to the latency of making changes, means that changes made to the live system can’t be automatically incorporated into the configuration source, and precludes the use of API-based tools for security remediation, cost optimization, operational runbooks, and so on. The combination of these factors and the lack of operator friendliness of the configuration format and change process make configuration drift and “breaking glass” during operations inevitable.

  • Virtually all generation-based tools, such as Helm, use a package model where explicit input parameters are exposed to the package consumer and the configuration generator is encapsulated by the package. This leads to multiple consequences, especially for off-the-shelf packages. (1) The demand to make packages flexible drives their abstractions to erode the more they are used, until virtually the entire configuration is parameterized and conditionalized. (2) The desire to keep the configuration DRY (Don’t Repeat Yourself) and handle all possible variants in one implementation increases the cyclomatic complexity. (3) The “abstraction” of each package is package-specific. The consequences of these are that the configuration generators tend to become very complex and are not amenable to the development of reusable tooling on top of them, unlike the resources they encapsulate. They obfuscate the underlying configuration. In addition to obstructing tool development, it also obstructs mass changes, which are important for central roles, such as security, platform, and infrastructure personnel. Imagine changing Kubernetes securityContext settings across hundreds of different helm charts, for instance.

  • Configuration templates / generators that are forked or generated independently by AI agents are hard to keep consistent.

  • Embedding configuration in generator code/templates also implies that it needs to be rendered/evaluated in order to fully validate it, and even to understand it. However, because of the difficulty of managing fully rendered variants, most users render configuration on the fly during deployment, which leads to unpleasant surprises. This also makes it difficult for AI agents to find their mistakes quickly and then correct them. To some degree, AI agents work by trial and error and rely on tests to provide a signal regarding whether what they generated is correct or not.

  • Because a single implementation can generate so many distinct variants, that leads to large blast radius when the generator templates/code need to be modified. Do you really want AI agents changing high-fanout templates that are so complicated you don’t understand them?

These drawbacks are what led us to represent configuration as data instead. Fully rendered, WET configuration data, with a 1:1 mapping between configuration and live resources under management. Cut out the code layer of complexity and obfuscation. Expose the data for reading, searching, validating, and writing it via an API. Versioning doesn’t require git, and data enables an alternative approach to managing variants.

Configuration as data in git

Suppose you accept that configuration is data. The natural next question is whether it still makes sense to store and manage that data in git.

Git and GitHub, GitLab, and other git services provide:

  • Versioning and undo

  • Change history

  • Review and approval workflow

However, the configuration data is structured, nested data. There are also inter-relationships between objects — it’s a graph. With many variants that have to be kept in sync across environments and regions. We want the System of Record to provide consistent, centralized visibility and control over the configuration, accessible to central roles, including non-developers. We want to read, search, extract values, validate, change, and replicate configuration through an API. And we want to track the provenance of every value, down to the field level.

Git was built for none of that.

More specifically:

  • Git deals with files and text, line by line. It doesn’t have mechanisms for dealing with structured data, such as JSON or YAML, which can affect diffs, patches, searches, and more. For example, good luck extracting the values of a “name” field in Kubernetes configuration from YAML files in git. String search is not going to cut it. You need to clone all the repos and run yq.

  • Every change is a many-step workflow, not an API: clone, branch, edit, add, commit, push, create PR, merge, tag, push tags. Then pull those changes into a variant, resolve conflicts, and repeat the steps to save the result. This can be seen in our the kpt guide for our installer.

  • The distributed model gets in the way. We want a consistent, centralized, multi-player view of the current configuration and changes to it, not separate clones that can be changed independently and merged together in unexpected orders and with potential conflicts.

  • The expectation of human review of all changes is not realistic when automated changes are being made, whether by AI or deterministic automation, such as to record changes from live systems. Rapid automatic validation is what’s needed, after every change, not deferred until after a commit, push, create PR, and then a CI pipeline run to render and validate.

  • Git is repository-oriented. Most configurations are not stored in repositories just by themselves, but there isn’t a very usable way to pull and change just a selected part of a repository — the existing mechanisms are partial clone and sparse checkout. Permissions are coarse-grained read/write at the repository level. You can’t grant permission to modify a single resource stored in the repository, for example. CI/CD pipelines are generally associated with repositories. Centralized management of many repositories and pipelines is also a challenge. Generally repositories are owned by many different teams, such as individual application teams. There are no standard APIs for discovering and creating repositories.

  • It’s hard to ensure the reliability and integrity of structured data, metadata, and entity cross-references when data structures are just written to files in git using non-transactional tooling, such as a collection of command-line tools rather than a control plane, especially when users can modify them out of band. It’s worse than exposing the raw database of a system. See, for example, the many criticisms of Terraform’s state file.

  • Git history is not immutable. The history can be rewritten by force pushes.

Note that I did not mention github.com’s recent availability challenges. That’s more of a last-mile delivery issue for GitOps tools, which can be solved by a more specialized distribution and caching mechanism, such as container images. One wouldn’t use github.com as a CDN, either.

Hopefully that explains why we chose a database. If one starts from first principles and requirements for what we’re trying to enable, the decision to use a database is obvious, just as we don’t store database tables in git as templates that generate CSV files.

Have you wished that you could build management tools that didn’t cause configuration drift or just provide recommendations for you, or an AI agent, to implement in configuration as code? Have you tried to make similar resource changes across many different helm charts across many repositories, or have you asked an AI agent to do that for you? Have you wanted better tooling to make changes across a fleet of clusters, as opposed to opening lots of PRs/MRs to change values files? Do you frequently need to break glass in production and then integrate changes back into git? Have you found your platform team to be a bottleneck in customizing configuration templates for developer teams?

Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Kubernetes series.