I recently wrote about why configuration belongs in a database.

Representing configuration as data is not as easy as it sounds. Yes, one could just ingest Kubernetes YAML into Elasticsearch as text, an object store, strings or byte arrays in Postgres, or somesuch. However, the configuration is structured data, containing nested maps and arrays.

Consider operations such as:

  • Querying deeply nested fields like spec.template.spec.containers[*].securityContext.capabilities.add

  • Changing deeply nested fields — e.g., add or remove a capability

  • Associative data lookups: get the current image tag for the main container of the backend Deployment in namespace cart

  • Inject secret management, logging, and/or service mesh sidecar containers into all workloads

  • Merge all environment-variable changes from the staging deployment into a production deployment while preserving production-specific differences

Keys and values that need to be matched may be spread across several non-contiguous lines of text. Some could also have multiple matches in the same document, especially in the case of multiple YAML documents in a single blob or file. Due to differences between similar but distinct variants, context-based text patches may fail to match.

These are reasons that document stores like MongoDB and other databases, including Postgres and MySQL, support storing JSON and/or JSONB. They can deal with structured data, with or without schemas.

YAML, of course, can be converted to JSON, or JSONL. However, that loses the properties of text that people expect, such as:

  • Preservation of map field order

  • YAML comments

Even if one decided that a normalized key order, such as sorted alphabetically, was a reasonable tradeoff, comments in configuration are critical for explaining why some properties are set the way they are. I would not want to lose that capability.

Even tools that change values in YAML files may not be comment-preserving. YQ is pretty good about preserving comments, but not all tools are. The kustomize kyaml library (not to be confused with the more recent KYAML format) and the Go YAML module deal with those issues when unmarshaling and re-marshaling YAML. Maps are represented in memory as ordered maps. And head, line, and foot comments are recorded as well as values. But that requires maintaining the configuration as YAML. I did actually find a YAML database, but it’s file-based and not widely used.

To enable representing configuration as JSON, I use two techniques:

  1. Comments are converted to fields, with keys of the form $comment$TYPE$FIELD, where TYPE may be head, line, or foot. Document head comments are represented as just $comment$head$ , with no FIELD. Capturing the correct positions of comments can be tricky in a number of cases, such as line comments on array field keys and foot comments, in general.

  2. In cases where a transformation may not be comment-preserving, such as after converting configuration to/from JSON or Protobuf, or when updating a resource from the live state, changes are merged into the configuration instead of just replacing the configuration.

As I mentioned in a recent post, the ability to merge configurations is an important primitive for configuration management, for many scenarios: preserving comments, preserving post-installation changes after re-rendering configuration, preserving customizations while updating from another source, propagating changes across variants, and more.

Kubernetes configuration has special rules for merging associative lists based on element values, which is called strategic merge patch. ConfigHub uses a similar mechanism to preserve overrides in variants without maintaining patches explicitly. Such associative matching is also useful in queries, such as to reference a value corresponding to a particular container by name or environment variable by name.

Not only does ConfigHub support Kubernetes YAML, but it also supports application configuration in several formats, including YAML and JSON, but also INI, TOML, Properties, and Env. It can generate ConfigMaps for these formats automatically, similar to comparable features in kubectl and kustomize. Partly inspired by the Augeas configuration editing tool, partly due to my compiler background, and partly out of necessity of solving the same problems as with Kubernetes YAML, these other “native” formats are converted to the same underlying representation for queries, functions, merging, and other operations on configuration data.

Schemas are necessary for validation and can be useful in other situations also, such as explaining the meanings of configurations, when generating configuration, and when rendering configuration (e.g., distinguishing fields from user-defined map keys). While schemas for Kubernetes resources can be looked up based on apiVersion and kind fields, other varieties of configuration would represented type information differently or may not have a notion of type. ConfigHub uses a well known metadata field, configHub.configSchema to represent this information. The field is stored in ConfigHub, but is stripped when wrapping configuration with a ConfigMap. When validating a configuration using a JSON schema, the field is used to match the correct schema, but is then stripped prior to validation. Representing the field along with the configuration data avoids the need to wrap the data with an envelope when serializing the data.

Some properties useful for managing configuration are not always represented in schemas, such as field behaviors. An example in Kubernetes resources is immutability. ConfigHub has its own mechanism for tracking such properties. I extracted immutable fields for Kubernetes built-in resource types from Kubernetes source code.

An additional complication is that while most of the configuration is structured data, not all of it is, and sometimes the structure may not be obvious:

  • Configuration embedded as multi-line strings. This is common in Kubernetes ConfigMaps that aren’t automatically generated, and can also occur in annotations and in infrastructure resources, such as IAM policy documents in AWS ACK resources. knot8 has a mechanism for accessing embedded values, for example. Sometimes the type of the embedded data can be inferred, such as from ConfigMap data keys or by attempting to parse it, but some formats can be hard to infer automatically and/or may be difficult to distinguish from one another accurately. The best case is when a particular format is expected in a specific resource field, because that can be treated as schema-like information.

  • Other strings may contain substructure, such as container images and host names, where parts of the strings may need to be operated on.

  • Multi-line text strings, such as embedded shell scripts in entrypoint commands, which may need to have changes merged using text-based diff and patch.

  • Command-line flags in positional container args arrays, which need to be merged based on matching flag names.

Hopefully this helps explain some ways in which configuration data is not just text, but is not just JSON, either. And I didn’t even go into versioning, metadata, relationships, and deployment status information.

Being able to navigate the structure of the configuration is critical to being able to operate on the data programmatically, rather than just punting to a human or to AI. The more of such cases that can be handled, the more deterministic automation can be utilized.

But it is also important to maintain key user expectations of configuration, especially comments, which is an often overlooked benefit of maintaining state outside live systems.

Have you tried to use tools to change values in Kubernetes YAML files and found that they removed comments? Or did you try to write a tool using Typescript or Python and encountered that problem? Have you tried to search configuration using text-oriented search tools or databases? Have you stored Kubernetes or other configuration in a database? If so, for what purpose? Have you tried to manipulate JSON as text? If so, did you run into the trailing comma problem? How often do you encounter git merge conflicts due to unrelated changes when changing YAML configuration? Have you been bitten by incorrect indentation of YAML?

Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Kubernetes series.