I previously wrote about validating Kubernetes configuration and enforcing policies after every change, prior to deployment.
Introducing a new validation check or policy when you have existing configuration is likely to find violations. To help roll out a new validation or policy check in a non-disruptive manner, we created an Initiatives workflow. It initially focuses on policies supported by Kyverno, including Kubernetes ValidatingAdmissionPolicy.
The workflow supports creating triggers initially in Warn mode, which will evaluate the validating functions and attach ApplyWarnings for failed checks, but not block apply/release of the configuration data.
It then reports configuration units that fail (and pass) the new check.
The violations can be fixed using ConfigHub functions, or by direct edits. Claude Code appears to like using yq.
Once all of the violations are fixed, the trigger can be changed to enforced mode, and the initiative can be marked completed, as in a kanban board.
It’s simple, but provides some structure around the process. We also created a video to walk you through it for the first time.
Kubernetes policy tools, including Kyverno, typically have ways to both report violations and to convert them to enforced checks, but those workflows don’t necessarily work when running checks in CI, prior to deployment, because that’s partly specific to how each CI tool works. Kyverno’s CLI also handles that differently than in-cluster admission control and report controllers. Such mechanisms also are different for every tool, and different tools cover different checks. Initiatives provides a workflow to do this through ConfigHub that can be extended to work across any validation and policy tools integrated using validating functions.
In the future I also plan to explore integrating some of the security and other management tools I built with the initiatives mechanism to make it easier to coordinate fixing the findings of those tools across multiple teams.
Do you have a way to orchestrate the introduction of new policy checks in CI? Does it work across multiple policy tools? Does it work across all repos containing Kubernetes configuration in your organization? Does it work across all Kubernetes configuration tools in use? How do you fix issues one environment at a time? How do you handle policy exceptions by environment or by workload?
Reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.
You could also try out ConfigHub, which is now in preview.
If you found this interesting, you may be interested in other posts in my Kubernetes series.