In honor of Kyverno’s graduation in CNCF, I thought I’d write a post about using Kyverno to vet changes to Kubernetes configuration prior to deployment.
First, what is Kyverno? Kyverno is a popular Kubernetes “policy as code” tool. It is integrated with Lens, Devtron, and a number of other tools. I previously mentioned Kyverno in one of my earliest posts, on state-based policy constraints, and in one of my most popular posts, on Kubernetes configuration linting tools.
Kyverno policies can be expressed as Kubernetes custom resources and, therefore, as configuration documents. That means they can be changed and applied like any other Kubernetes resources, such as with GitOps tools like ArgoCD or Flux, and are easy to discover through the Kubernetes API. They don’t have to be pushed to or pulled from another storage location.
These policies can include fairly arbitrary expressions written in CEL to test values of Kubernetes resources, in effect imposing constraints on Kubernetes resources accepted as valid. This allows you to ban configurations that Kubernetes may allow, but that you don’t want to allow in your cluster(s) or at least in some namespaces. This differs from RBAC authorization by constraining WHAT changes are made rather than by constraining WHO made the changes.
For example, here’s a policy that prohibits the use of the :latest tag in Deployments by checking the value of the appropriate field of the resource (slightly simplified compared to the sample):
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
name: disallow-latest-tag
spec:
validationActions: [Deny]
matchConstraints:
resourceRules:
- apiGroups: ['apps']
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [deployments]
validations:
- expression: >
object.spec.template.spec.containers.all(c,
!c.image.endsWith(':latest')
)
message: "Using 'latest' tag is not allowed."
Kyverno has a sizable policy library, and of course you can write your own, as well.
Normally, these policies are enforced by Kubernetes dynamic admission control. However, one then doesn’t find out until applying the configuration whether it passes the policy checks or not. If using ArgoCD or Flux, this failure may show up as a deployment that never successfully fully synchronizes with the cluster. It would be better to find out about such problems earlier and in a more obvious way.
The typical approach to checking the policies in advance would be to render the Helm charts, kustomizations, cdk8s constructs, etc. to plain Kubernetes YAML, and then use the kyverno CLI to apply the policies to the output, generally in CI.
% kyverno apply disallow-latest.yaml --resource=./rendered-resources/
Applying 1 policy rule(s) to 1 resource(s)...
policy disallow-latest-tag -> resource default/Deployment/latest-deployment failed:
1 - Using 'latest' tag is not allowed.
pass: 0, fail: 1, warn: 0, error: 0, skip: 0
Because the policy specifications are likely owned by a central team, such as a platform team or security team, rather than by the application team, and they are applied to multiple applications, they are typically in a different git repository. Fortunately, the kyverno CLI supports that (private repo support):
% kyverno apply https://github.com/kyverno/policies/tree/main/best-practices-vpol \
--resource=./rendered-resources/
This works, but there are multiple steps between making a configuration change and finding out that there was a problem with the change, results in CI are buried in log files unless they are pushed somewhere, and for teams that own their own CI pipelines/workflows, the policies are opt-in, and must be added to every repo’s CI pipelines/workflows and kept up to date.
I integrated Kyverno with ConfigHub to demonstrate how Kyverno and similar policy tools can be leveraged to prevent misconfiguration by validating the configuration after every change.
First, a little background on ConfigHub. In ConfigHub, configuration is always fully rendered, or WET — configuration is represented as data. In the case of the Kubernetes Deployment that failed the policy check above, it would look as follows, with no variables, conditionals, loops, or other computation:
apiVersion: apps/v1
kind: Deployment
metadata:
name: latest-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- name: test-container
image: nginx:latest
Configuration is stored in units in ConfigHub rather than as files in git. You could upload the configuration above into a unit with:
% cub unit create latest-deployment latest-deployment.yaml
(There’s a basic tutorial to get started with ConfigHub. If you’re using Helm or kustomize with ArgoCD or Flux, then we can import and render the configuration automatically, but that’s a topic for another post.)
ConfigHub has a capability analogous to Kubernetes dynamic admission webhooks called Triggers. Triggers invoke functions automatically on each configuration change. A function is code that executes on configuration data that ConfigHub can invoke. Some standard functions are built into ConfigHub. You can also write your own functions and build them into workers.
A worker is analogous to a controller in Kubernetes or a runner in CI (though it has specific functionality built in rather than executing arbitrary containers). You run it, typically in your Kubernetes cluster, and it connects back to ConfigHub and performs operations on your behalf.
As an example, I created a custom worker that calls Kyverno’s admission webhook to evaluate policies on configurations in ConfigHub.
To try it, clone our sdk:
% git clone https://github.com/confighub/sdk
The code is in [examples/kyverno-server](https://github.com/confighub/sdk/tree/main/examples/kyverno-server), which also contains a demo script that creates a kind cluster, installs Kyverno and some policies, starts the custom worker, and invokes the function imperatively, as follows:
% cub function do vet-kyverno-server --unit latest-deployment \
--worker "$KYVERNO_WORKER"
Validation functions return a pass/fail result and details about issues found in the case of failure.
false 0 policy "disallow-latest-tag" rule "validation": Using 'latest' tag is not allowed.
Attributes:
<nil> default/latest-deployment apps/v1/Deployment
To create a trigger using this function, you can do:
% cub trigger create kyverno Mutation Kubernetes/YAML vet-kyverno-server \
--worker "$KYVERNO_WORKER"
And then when changes are made to the configuration data, the function would automatically be invoked, and if it failed then an apply gate would be attached that would block applying the configuration to the cluster until the issue was addressed.
This approach enables shifting Kubernetes admission control left, to the point where the configuration is modified in storage, prior to deploying changes to the cluster. Any admission controller you have registered with your cluster to be invoked on Kubernetes API calls can also be invoked ahead of time, by your worker, through ConfigHub. Because the worker calls back to ConfigHub, ConfigHub does not need access to your cluster. However, if you’d prefer that the worker didn’t call your admission webhooks, check out the other Kyverno worker example, which executes the Kyverno CLI with specified policies and resources similar to the examples above.
These Kyverno worker functions are just a couple examples of the kind of validation and policy checks that can be enforced centrally using ConfigHub. There’s also a kube-score example, as an example of integrating a configuration scoring tool. For schema validation, kubeconform is integrated as the vet-schemas function. We plan integrations with Kubernetes validating admission policies, Open Policy Agent, and other tools in the future. Additionally, you can use CEL expressions (vet-cel) or Starlark (vet-starlark) to write policy checks without building and running your own worker for that purpose.
Whether you are making a change to a configuration by hand, with yq or another tool, with a ConfigHub function, or with an AI agent, with ConfigHub, not only is the change not obfuscated by layers of configuration generation code, but it can also be immediately automatically vetted, thereby reducing the opportunity for misconfiguration.
How have you tried to prevent common mistakes when people in your organization change Kubernetes configurations? Do you render your configurations and evaluate Kyverno or other policies in PRs before the changes are merged? Is there anything you would like to improve with that process? Do changes ever bypass the policy checks?
Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.
You could also try out ConfigHub, which is now in preview.
If you found this interesting, you may be interested in other posts in my Kubernetes series.