I previously wrote about abstraction being the wrong way to simplify configuration. What I really meant was that unidirectional configuration generators were the wrong way to implement abstractions.

Internal developer platforms and other application platforms built on Kubernetes present simplified abstractions that don’t expose all Kubernetes functionality, but that also decreases flexibility and couples operational concerns and security concerns with development concerns. Many such abstractions are also not as widely used standards as the Kubernetes APIs.

One application abstraction gaining a fair bit of adoption is Score. Score looks a little like Docker Compose, but also contains some prominent Kubernetes features, such as probes and resources. The score-k8s tool will generate Kubernetes resources from a Score spec. The documentation says “Iterate by changing the score files, and re-running generate.” That’s the typical unidirectional generation approach.

Since the Score spec doesn’t cover all operational concerns, they must be added somehow. The documentation recommends patching the generated configuration using kustomize or built-in JSON patch operations. That’s quite a few patches: replicas, security context, pod anti-affinity, sidecar containers, various controller annotations and labels, …

When using ConfigHub, ConfigHub is the authoritative record of the configuration data. The resources don’t need to be patched over and over again on the fly. Instead, functions and tools can be used to customize the configuration, and those changes are persisted. However, that means we don’t want some external files to be persisted as the source of truth.

Instead, we want to regenerate the Score spec from the data in ConfigHub. I built a tool to do just that, k8s-to-score. Point it at a space and it identifies the workloads and generates Score files for them. For example:

% k8s-to-score --space nonprod-c3agent --out-dir /tmp/agent-score

One of the components it generated looks like this:

apiVersion: score.dev/v1b1
metadata:
  name: c3agent-gateway
containers:
  gateway:
    image: c3agent-gateway:sha-eaedf4dfa33ef25dd1e8b06e2f31bd2d007ce412
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 10m
        memory: 128Mi
    variables:
      CONTROL_PLANE_ADDR: c3agent-controlplane:50050
      CP_GRPC_PORT: "50050"
      CP_HTTP_PORT: "8080"
      CP_SCHEDULE_INTERVAL_SECS: "7"
      DEFAULT_MAX_BUDGET_USD: "10.00"
      GATEWAY_ADDR: c3agent-gateway:50052
      GW_GRPC_PORT: "50052"
      MAX_CONCURRENT_TASKS: "5"
      TASK_NAMESPACE: c3agent
      TASK_PVC: c3agent-taskdata
service:
  ports:
    grpc:
      port: 50052

This obviously doesn’t reflect all of the details in the Kubernetes Deployment, Service, Ingress, PodDisruptionBudget, HorizontalPodAutoscaler, and other related resources. However, when score-k8s generate is run on this, it roundtrips to a subset of the original resources.

If the Score spec is changed and then the manifests are regenerated, the output can be merged back into ConfigHub’s full representations of the resources. For example, if the cpu request is changed to 100m, that’s also updated in the output:

...
            requests:
              cpu: 100m
              memory: 128Mi
...

And that change can be merged back into the copy of the resource in ConfigHub.

ConfigHub supports several patterns where configuration changes can be merged into other units, including merging changes from external sources. The installer supports this, as well, so that configuration can be re-rendered if inputs or other choices or facts change, and then the changes in the rendered output can be merged back into ConfigHub.

Through this approach the fully rendered configuration data can serve as the authoritative record, with the external platform abstraction as just one temporary, partial view of the resources. The abstraction can then interoperate with other users, agents, functions, or tools that modify the resources. For example, operators can use views and tools that expose details that are relevant to their responsibilities.

This kind of interoperability is typical and expected of API-based tools. For example, one expects to see changes made with a CLI through a web UI for the same service, and vice versa. However, this is not the case with configuration as code — changes made through other tools cause configuration drift. In this case, all changes are being made to the same authoritative copy of the configuration data, so there is no drift.

Configuration as code has made us accustomed to sacrificing the benefits of APIs in the name of repeatability (i.e., the ability to create additional variants). That tradeoff was expedient, but unnecessary, and suboptimal. I hope to see wider adoption of view-based abstractions in the future, as opposed to just generator-based abstractions.

Have you run into the challenge where some team needs one more API feature exposed through your platform abstraction, and then that happens repeatedly for different features? How did you handle that? Did you expose them and erode your abstraction? Have you found developer-oriented abstractions to hamper operational changes due to the use of configuration generators like Score, Helm, or Crossplane compositions? Have you built platform abstractions using the view/projection approach rather than generators? If so, were they built directly on Kubernetes APIs, or on top of configuration files in git, or on top of another storage system?

Reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in my other posts.