The concept of a "golden path" is well popularized1 in Platform Engineering: Platform engineers build tools that help other software teams quickly set up specific types of applications or stacks while at the same time following organizational best practices.
It's a good idea, but it has a number of pitfalls.
- First, the tool can be overly simplistic and not meet the needs of its customers. They leave it on the shelf and never adopt it.
- Second, the tool can become overly parameterized in order to meet all the different requirements coming from its customers. It becomes as complex as the underlying infrastructure and just gets in the way without simplifying anything.
- Third, the tool can become a victim of its own success. Dozens or hundreds of workloads are created with the tool. Then the tool eventually needs to evolve or be replaced. But now that might break or orphan all these running workloads.
These pitfalls are addressed by taking a configuration-as-data approach:
Configuration-as-data means that configuration is stored in a database, just like any other business data (customers, employee data, transactions, products, etc.). Unlike infrastructure-as-code and the various templating and config-generator approaches, config data has its own lifecycle and "home", separate from the tools. It is not just an ephemeral output of a processing pipeline.
In this world, Golden Path tools operate on the data but they do not own it. The data lives separately from the tool. The tool reads and writes with the expectation that the data is touched by other tools and that, one day, the tool may be replaced by other tools while the data remains.
Importantly, the data uses the exact schema of the underlying technology (Kubernetes, AWS, GCP, etc). No new schemas are invented. An EKS Cluster is just what AWS defines it to be. Set spec.logging.clusterLogging.0.enabled to true and you have control-plane audit logging at any time regardless of what the Golden Path tool supports. A Kubernetes deployment is stored and accessed in its full glory. Flip automountServiceAccountToken after the fact even if the Golden Path tool doesn't expose it.
To demonstrate this, I created an example Golden Path tool to set up an LLM inference stack on an AWS EKS cluster. Let's walk through this example at a high level.
The EKS Inference Golden Path tool
I highly encourage you to follow the README in the eks-inference repo and experience all this for yourself, but I know there is not always time for that so I'll briefly explain how it works.
The EKS inference tool (eksinf) uses ACK to provision AWS resources. ACK needs to run in a Kubernetes cluster itself, so everything starts with booting a local Kind cluster. Next a credential is needed in order to access AWS resources and then the AWS infrastructure is provisioned.
Once the EKS cluster is up and running with ArgoCD installed, it gets enrolled into ConfigHub so the next layer of services can be provisioned while managed in ConfigHub. This includes an NVIDIA GPU plugin and Karpenter for nodepool management. So far, the total cost footprint is EKS, 2 t4g.mediums and a NAT gateway - ~$0.20 / hour.
Next, an AI workload consisting of a quantized Qwen2.5-7B model being served by vLLM can be scheduled. This step will trigger Karpenter to provision a node to provide the GPU resources needed and that bumps up the cost. It can be quickly reversed by scaling the AI workload to zero which causes Karpenter to shrink the nodepool to zero after a bit. The GPU node runs roughly $0.80–1.00 per hour depending on what Karpenter finds available.
When the LLM is up and running, you can talk to it. The stack ships a small terminal chat client that runs inside the cluster, reachable with kubectl exec.
The repo itself has a lot more details for those interested.
The Configuration-As-Data difference
A full stack like this EKS inference example has countless configuration options ranging from the VPC network layout to EKS authentication configuration, Karpenter nodepools, LLM model config, etc.
The cub eksinf tool only has a few commands with a few flags each. It picks a bunch of defaults for you. That's the value of a golden path tool and it works great right up until you need something slightly different.
For example, this tool sets consolidateAfter on the quantized GPU nodepool to 5m. Scale the workload to zero and the pod terminates immediately, but the GPU node lingers for another five minutes — about a dollar an hour to do nothing.
A user of the eksinf tool will likely get frustrated by it and want the tool to be more configurable. But this is just one of many configuration options that users will want which either leads to the golden path tool becoming inadequate or drowning in options and flags.
With configuration-as-data, this is not the case. Now I can simply edit the config directly when I need to. For example, I can use a built-in mutating function like this:
cub function do --space karpenter-dev --unit nodepools \
--where-resource "ConfigHub.ResourceName = '/quantized-gpu'" \
set-string-path karpenter.sh/v1/NodePool spec.disruption.consolidateAfter 1m
I can also manually edit the yaml in my default editor with:
cub unit edit nodepools --space karpenter-dev
I know this might seem a bit odd if you are new to ConfigHub or configuration-as-data, so let me explain what's happening here. All along the way, the actions performed by the eksinf tool resulted in configuration data being written to ConfigHub. It didn't touch the infrastructure APIs directly. The infrastructure is controlled by the data, not by eksinf. So you are always free to jump in and modify that data and re-publish it. The eksinf tool is not authoritative. The config data is. That's where the magic lies.
The key ingredients
First of all, you need a data store like ConfigHub that is purpose-designed for configuration rather than adapted to it. Concretely, that means a few things. Changes address individual fields — functions like the set-string-path above set one path in one resource, so two tools editing different fields of the same Unit are not in conflict. Queries select units by what is inside them, so "every nodepool with a GPU taint across all my clusters" is a filter rather than a script. Every change produces a revision you can inspect, compare and roll back to, and related changes can be grouped into a ChangeSet and reverted together. And a change can be applied across a fleet as a bulk operation instead of a loop.
You also need to write your tooling with this approach in mind. Tools don't own the data. They mutate data in place and must not clobber or fail when data is changed by other sources. This is not as difficult as it might seem. It mostly consists of ensuring idempotence and addressing individual config data fields directly such as the consolidateAfter field in the example above. ConfigHub ships with a large library of built-in functions to make this easy.
Once you have multiple tools and tasks operating on the same data, you need to make the right decisions about when a tool gets to overwrite past changes. This is what merging is for. ConfigHub records which change last set every individual value — a per-path record of who touched what — so an incoming change can be applied without stepping on a value something else already set.
That per-path record is what keeps your edits intact by default, and you can mark any path as explicitly off-limits with cub unit set-protection. Some of these concepts are familiar from standard version control tools like git, but a line/byte oriented tool like git is suboptimal compared to a tool designed specifically to handle versioned data structures like config data.
Golden path tools that work
Thanks to the "escape hatch" of having full access to the underlying config, the EKS inference tool is useful as it is without having to respond to a barrage of feature requests for additional flags and options.
The tool will of course still benefit from incremental enhancements, but it will not be sitting on the shelf just because it is missing a feature. Users can contribute to the tool or build their own tools and use them in parallel. One day, when the tool is showing its age, it can be thrown away and replaced without any risk to existing deployments.
I picked LLM inference for this example because it is an exciting new type of stack with plenty of moving parts. But the same approach works for any kind of deployment pattern. You can build your own tool on ConfigHub for a web app stack, bot stack, data processing stack, an application framework, a platform layer, and anything else you can imagine.
Try it
Head over to the eks-inference repo to try it out. If you just want to see the shape of it, cub eksinf sandbox up builds the entire configuration — every Space, Unit, link and release — and creates no infrastructure at all. It takes under a minute, needs no AWS account, and costs nothing. There are also several documents in the repo that might interest you, for example, it makes use of "Helm Chart flattening" which helps address the mismatch between Helm as a package install tool and declarative cluster management.
If you want the real thing, the walkthrough in the README stands a full stack up and back down again. It runs about $130/month if you leave it up (which you probably don't want to do). Teardown is part of the walkthrough and tries hard to ensure all resources are deleted. But please double check your AWS account when you're done, just to be safe. One command that tells you what the whole account is costing per day:
aws ce get-cost-and-usage \
--time-period Start=$(date -u -v-1d +%F),End=$(date -u +%F) \
--granularity DAILY --metrics UnblendedCost \
--query 'ResultsByTime[0].Total.UnblendedCost.Amount' --output text
Cost Explorer has to be enabled on the account, the data lags about a day, and each call costs a cent. On Linux, use date -u -d yesterday +%F.
Either way, go and edit something the tool never intended you to edit. That's the part that makes this different and makes it work. Publish it, watch it land, then run cub eksinf install again and watch your change survive.
I am usually hanging out in our Discord and would love to hear feedback and answer any questions.
ConfigHub Signup is open to anyone.
Hero image generated with Gemini.