I previously wrote about some Kubernetes namespace policy management tools I built on ConfigHub: namespace-manager, rbac-manager, network-policy-manager. Next, I built some workload-oriented management tools.

Workloads have some interfaces that are specific to them, such as command-line flags, environment variables, application configuration, probe endpoints, and so on. The configuration of those properties is best performed with the schemas and constraints for those interfaces in hand. I’ll write another post about that.

Workloads also have security and operational aspects that are cross-cutting and at least partly determined by the deployment environment, such as operating-system capabilities, computational resources, scaling automation, network access, and observability integrations. The workload may have some requirements of these aspects, but their configuration is not unique to the workload and is often at least partly determined by factors outside the workload implementation, such as the choice of autoscaling tool or observability tool, as well as the application or infrastructure platform, which is Kubernetes for this discussion. Typically the security posture is determined at the organization level, as well, or at least at a larger scope than just one workload.

With configuration as data, not all changes to configuration need to be implemented in a single monolithic generator. They can be decoupled and performed by interoperable tools, just as when building tools against APIs.

To demonstrate this, I built a suite of tools, in one day, each with its own agent skills:

All of these tools can enforce desired postures, as well, by installing ConfigHub Triggers via guardrails commands.

As with the policy tools, these tools can provide lists of the relevant resources across clusters, snapshot summaries, and detailed lists of findings.

For example, from the workload manager:

CLUSTER                                 WORKLOADS  PDBS  UNITS  GATED  UNAPPLIED
dev-cluster                             19         0     33     0      21
prod-cluster                            19         0     38     0      24

2 clusters, 38 workloads, 0 pdbs, 71 units (0 gated, 45 unapplied)

SEVERITY  ANALYZER   CLUSTER          NAMESPACE  KIND         NAME     MESSAGE
...
HIGH      security   prod-cluster     appvote    Deployment   vote     container "vote" does not set runAsNonRoot: true
HIGH      security   prod-cluster     appvote    Deployment   worker   container "worker" allowPrivilegeEscalation is not false
HIGH      security   prod-cluster     appvote    Deployment   worker   container "worker" does not set runAsNonRoot: true
...
MEDIUM    probes     dev-cluster      appchat    Deployment   backend  container "backend" no readiness probe
MEDIUM    resources  dev-cluster      appchat    Deployment   backend  container "backend" no cpu limit
MEDIUM    resources  dev-cluster      appchat    Deployment   backend  container "backend" no cpu request
MEDIUM    security   dev-cluster      appchat    Deployment   backend  container "backend" readOnlyRootFilesystem is not true
...
LOW       hygiene    dev-cluster      appvote    Deployment   redis    container "redis" terminationMessagePolicy is not FallbackToLogsOnError
...
39 workloads (0 pass, 21 warn, 18 fail)
...
  dev-cluster/appchat Deployment/backend:
    [hygiene] container "backend" terminationMessagePolicy is not FallbackToLogsOnError
    [probes] container "backend" no liveness probe
    [probes] container "backend" no readiness probe
    [resources] container "backend" no cpu limit
    [resources] container "backend" no cpu request
    [security] automountServiceAccountToken is not false
    [security] container "backend" allowPrivilegeEscalation is not false
    [security] container "backend" does not drop ALL capabilities
    [security] container "backend" does not set runAsNonRoot: true
    [security] container "backend" readOnlyRootFilesystem is not true
    [security] container "backend" seccompProfile is not RuntimeDefault
...

(terminationMessagePolicy is a pet feature of mine. I find it useful.)

If this were a typical policy tool and you were using helm, cdk8s, tanka, or other configuration as code tool, then you’d have to track down the right repo and configuration files, render the configuration and feed it to the policy tool, ask your AI agent to implement fixes for the correct environment(s), review the changes, and render the configuration again, for all potentially affected environments, and feed it to a validation tool.

But this is configuration as data, so those checks ran against fully materialized configuration and you can just change the configuration directly with the tools.

Want a PodDisruptionBudget?

% ./bin/cub-workload ensure-pdb appchat-prod/frontend --max-unavailable "10%" -o table
Dry run — would create PDB Unit appchat-prod/frontend-pdb:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
  namespace: appchat
spec:
  maxUnavailable: 10%
  selector:
    matchLabels:
      app: frontend

Re-run with --commit --change-desc "…" to create the Unit. It is not applied until you apply it.

That automatically matched the namespace and labels of the specified workload, similar to kubectl expose, which makes it even simpler than kubectl create pdb. For a single workload, an AI agent could also author the resource from whole cloth, especially if it had a sample to start from, as in our agent skills or in our package of resource samples.

Want spreading via pod anti-affinity? ./bin/cub-workload ensure-spread appchat-prod/frontend. CPU and memory resources? set-resources. Probes? set-probes . Security context and disabled service account token? harden . There are lots more features I don’t have space to show in this post. There’s no need to write YAML by hand when tools are perfectly capable of doing it. Remember, these tools are operating on automatically versioned storage, not operating on the clusters directly.

To be agent-friendly, the commands execute in dry-run mode and output JSON by default. Agents can execute the commands and generate summary tables regarding the current state and changes they made. With approval, they can “commit” changes by performing the same operations without dry run. The changed configuration will be validated automatically, and humans can still review and approve the changes before releasing them to the clusters, and can revert or adjust them as needed.

I’m also experimenting with writing higher-level intent to the resources using annotations. I’ll write about that in a future post once I have more experience with it.

Under the hood, the tools work through a combination of ConfigHub filters, functions, such as set-container-resources and set-pod-container-security-context-defaults, client-side analysis, client-side mutations using the function SDK, saved function invocations, triggers, and other ConfigHub features.

Because the configuration data is authoritative, these tools can interoperate with other tools that read and write the configuration data, including other ConfigHub functions, such as set-container-image-reference and set-replicas, and AI agents editing the configuration. Tools can be single-purpose or multi-purpose, as desired, but they don’t need to be monolithic, covering every field of Kubernetes Deployment, StatefulSet, and DaemonSet resources. The tools can also be versatile. They can analyze, summarize, generate, backfill, promote, validate, and enforce.

And because we store configuration in a database instead it being sprawled across many git repositories owned by different teams, the tools don’t need to clone lots of repositories, make ad hoc edits, commit and push changes, and create lots of PRs. Instead, they can call APIs. I’ll write another post about the “initiative” system that we’re working on for changes that span multiple teams.

I’m sure the tools still have rough edges and need more work to make them usable — I’ve barely had time to try them yet, but the point is that we CAN build and use tools if we’re not using configuration as code. The tools can operate on variants of the same application across clusters, and across multiple different applications, not just a single application, using standard APIs. Composable, interoperable tools are more reusable than all-encompassing templates.

That last point is pretty important. Yes, for one application, you could use a heavily parameterized template library to create, say, a HorizontalPodAutoscaler, and add some values to drive it. But every Helm chart exposes different input values, so there’s no standard API that one could build tooling around.

The common alternative automated mutation methods are less transparent and more complicated operationally:

  • kustomize overlays

  • mutating dynamic admission control

  • Operators (e.g., to inject sidecar containers or create PDBs)

ConfigHub makes it possible to directly modify configuration while maintaining multiple similar-but-different variants of the same configuration through mechanisms that help keep them in sync. That’s one of the key challenges we’re solving with ConfigHub.

I think the approach of building tools on top of configuration as data has a lot of promise. Try the tools and let me know what you think. Or build your own and let me know how they turn out.

What cross-cutting customizations do you frequently need to make to Kubernetes workload configurations? What are common changes that aren’t supported by helm charts you use? How do you override them? What common operational changes do you need to make beyond just replicas and resources? How often do people in your organization “break glass” and change clusters directly? What are the reasons? Have you migrated from HPA to KEDA, or Ingress to Gateway, or Nginx Ingress to Traefik? Do you have operations, platform, or security teams that need to modify configurations owned by application teams? What kubectl commands or other tools should we adapt to ConfigHub?

Reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Kubernetes series or my IaC / declarative configuration series.