One of my main original rationales for representing configuration as data was the inability to build tools on top of configuration as code. Instead, almost all tools that work with Kubernetes resources call its API directly. If they need to make changes, they either cause drift or just make recommendations, which then need to be implemented in configuration code.
That is a big opportunity cost with the path the industry has taken with config as code and GitOps, especially now that it’s so much easier to build apps with AI agents. Every time I see a new Kubernetes dashboard that doesn’t provide the ability to write to the cluster because “ClickOps is bad” or “changes should go through git” I see it as a sign that the approach is fragmented and incongruent.
Today, a tool tells you what the problem is and then you or your AI agent needs to track down the right part of the right git repository (or, worse, lots of repos), clone, branch, edit a template and/or input values, commit, add, push, open a PR, and wait for CI to render the configuration and run validation checks on it, if you’ve set them up.
There’s a ton of friction in this process: context switching, complexity, toil, blast radius, latency, … Do friction, toil, and complexity matter, now that AI agents can bear the load? Does building tools in the era of AI agents even still matter?
Good questions. Why build tools?
Unlike generic configuration as code formats, tools can encapsulate logic encoding domain expertise, API syntax details and semantics, and non-trivial workflows. And can do it in a manner that is decoupled from configuration data and reusable across different configurations. They don’t all have to be based on the exact same Helm chart, for example. A “tool” could also be a feature of an internal developer platform.
You may wonder whether AI models have the necessary expertise to perform all necessary tasks. Increasingly often they do, or could by looking up the right information, but leveraging that expertise is not easy.
As someone who uses AI every day and who has tried to create AI agent skills, I find that predictability and repeatability are big challenges still, even with the latest frontier models. For specific workflows, especially personal workflows, it generally works better to ask the AI agent to write a script or some kind of workflow representation if you need to be able to repeat the task. However, scripts and workflows can be hard for people who didn’t write them to understand and can also be brittle, depending on how much work went into creating them. Tools can embody deterministic code that can be tested thoroughly and documented, and then executed the same way repeatedly by all organization members.
Discovery is another reason. Not everyone who wants to perform a task may know all the details they need to provide to an AI agent in order to perform the task correctly, or how to point it at the best sources of knowledge for the task. A lot of work is being done around skills, but from what I can tell, automatic triggering of skills doesn’t work. When the description is shorter, it doesn’t match. When it’s longer, it would fill up the context so the agent defensively decides to ignore it. Discovery is one of the motivations for developer portals.
Even when a skill is invoked explicitly, parts of it may be ignored, or could become out of date due to the difficulty and cost of continuously testing skills. Trying to make skills work consistently feels like whack-a-mole: fix one issue and then the model hallucinates something else or takes an unexpected turn. The behavior is also different when using different models.
Even when the interface surface is targeted at AI agents, such as using MCP, there can be valid reasons to create tools with narrower interfaces, including efficiency (in tokens), permissions, blast radius, and general safety.
Also, Kubernetes and cloud configuration contain a lot of details. I have found that generating and even editing configuration can require many iterations of questions and answers (e.g., using the AskUserQuestion tool), clarifications, additions, removals, fixes, etc. It works best when the agent has a good starting point (e.g., a vetted reference) and can be provided with data that answers most questions (I’ll write another post about that), but I found that using tools, ConfigHub functions in this case, can more efficiently (and reliably) set standard best-practice properties like the securityContext.
So let’s say we want to use AI to build a tool. How do we do it?
With configuration as data, configuration can be changed through an API.
As a demonstration, in about 3 hours, including the time to produce a PRD, design document, and implementation plan, I used an AI agent to build an RBAC manager application on top of ConfigHub. (Keep in mind that I am neither a frontend engineer nor a UX designer and this was just a quick example.)
Because the RBAC [Cluster]Roles, [Cluster]RoleBindings, and ServiceAccounts are in ConfigHub, the tool can analyze them. (And if they weren’t, they could be imported.) The screenshot above shows risky and overly broad permissions, as an example. It can also identify missing roles:
And query who can access what:
That’s all possible because the configuration is already fully rendered, WET YAML. It doesn’t need to be rendered. It also spans multiple clusters. But that is just readonly functionality so far.
The big difference is that if I want to address any of the issues that are discovered, I can make a change, and review it before it is applied. Approval can also be required, depending on the environment.
ConfigHub also supports versioning and policy enforcement. The capabilities that would normally drive someone toward configuration as code have been addressed.
Is it possible to build this type of application on top of git, as I’ve seen often suggested? Not entirely. I have tried. Definitely not on top of Helm charts, kustomize patches, cdk8s code, jsonnet code, or the like. I’ll write another post about why ConfigHub stores configuration data in a database.
Hopefully that gives you a sense of what’s possible with a change of approach, by treating configuration as data. Instead of writing templates and creating rigid abstractions, build apps. Take your favorite AI agent for a spin and give it a try.
Should I transform the RBAC manager into a Backstage plugin? Would you build a tool with a MCP interface instead? Should I add support for other resource types, such as Namespace, NetworkPolicy, or ResourceQuota? If so, which ones would be most useful? What Kubernetes management application would you build? A securityContext manager? An API version migration tool? A tool to add HorizontalPodAutoscaler, PodDisruptionBudget, or Prometheus ServiceMonitor resources to applications? To manage common kinds of sidecar containers? To allocate CIDR blocks? To create and manage EKS clusters using ACK?
Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.
You could also try out ConfigHub, which is now in preview.
If you found this interesting, you may be interested in other posts in my Kubernetes series.