I previously wrote about integrating Kyverno with ConfigHub in order to evaluate Kyverno policies after every configuration change and block deployment in the case of violations.

The purpose was to find policy violations and configuration errors earlier, especially prior to deployment. As I mentioned, it is possible to do this by rendering configuration and running validation checks in CI, but that requires many more steps (e.g., the git commit and push dance, rendering), can be slow (minutes), and can be hard to manage centrally.

I encountered another issue with the CI approach: AI agents. AI agents are getting better and better at generating and changing Kubernetes YAML, but they still make mistakes and they also aren’t aware of organizational policies and conventions. Even with AI agent skills, I have found agents to not 100% reliably follow instructions from documents, so that is not an adequate method of conveying policies. That said, skills do have their place, and I created some agent skills for ConfigHub.

AI agents don’t know what they don’t know and work by trial and error to some degree. They can iterate (“hill climb”) on their own when their work is verified, can iterate faster when the verification is faster, and consume fewer tokens when the output is concise and in a consistent format, as opposed to typical logs and human-oriented tool output.

So, I integrated more validation, scoring, and policy tools with ConfigHub.

So far I implemented support for:

These integrations convert the various tool output to a consistent JSON data structure containing messages from the tools, resource types and names, failing configuration paths where reported, and scores where provided. Each example includes a demo script so you can see them in action.

When used in triggers, failures automatically attach corresponding Apply Gates to the configurations. It’s also possible to set the trigger to just warn instead of blocking deployment.

With details regarding the failure.

The apply gates and warnings are reported by the API and can be accessed by the CLI as well, for use by AI agents. The triggers can be invoked on demand, as well, to return the full responses.

Other than kubeconform, which is integrated as a library, some of these tools are CLIs and some are admission webhooks. Kyverno supported both, so I integrated both. The CLIs are pretty slow — up to multiple seconds, even without spinning up a container. So a server is a more viable way to provide interactive performance, though it does cost something to run, unless it can scale to zero, which would also incur some cold-start overhead.

I was asked a good question on the Kyverno post about how integrating with admission webhooks was different from kubectl apply --dry-run=server --validate=true. A few differences:

  • Cluster credentials aren’t required

  • It provides more control over what checks are performed, and eliminates other potential causes of failure, which I wanted to report separately

  • Failure details are easier to extract

It’s also possible to implement your own policies and checks in CEL, Starlark, and Go. The RBAC manager uses CEL. There are also some additional built-in validators, such as to check whether there are duplicate merge keys (e.g., environment variables) and whether immutable fields are changed.

Once I created a validation function that implemented a particular pattern, I found AI pretty good at applying the pattern to a different tool or check.

I just started to iterate on the agent skills and am interested in your feedback on what you’d like to use them to do with ConfigHub. I used the triggers-and-applygates skill on the demo data shown above in Claude Code.

Here's the full diagnosis of initiatives-demo — 19 Units.

  Policy in force — 5 vet-kyverno Triggers (worker kyverno-cli-worker)

  ┌─────────────────────────────────────┬───────────────┬──────────────────────┐
  │               Trigger               │     Tier      │ Currently firing on  │
  ├─────────────────────────────────────┼───────────────┼──────────────────────┤
  │ image-registry-restriction-check    │ Gate (blocks) │ 8 Units              │
  ├─────────────────────────────────────┼───────────────┼──────────────────────┤
  │ disallow-host-ports-check           │ Gate (blocks) │ 0 Units              │
  ├─────────────────────────────────────┼───────────────┼──────────────────────┤
  │ run-as-nonroot-enforcement-check    │ Warn          │ ~all 17 warned Units │
  ├─────────────────────────────────────┼───────────────┼──────────────────────┤
  │ liveness-and-readiness-probes-check │ Warn          │ 14 Units             │
  ├─────────────────────────────────────┼───────────────┼──────────────────────┤
  │ resource-limits-enforcement-check   │ Warn          │ 2 Units              │
  └─────────────────────────────────────┴───────────────┴──────────────────────┘

  Tier 1 — Blocking gates (8 Units, apply refused)

  All 8 are blocked by the same gate, image-registry-restriction-check (requires images to start with ghcr.io/acme/):

  aichat-postgres, aichat-redis, docs-search, eshop-postgres,
  eshop-redis, kyverno-cli-worker-unit, portal-postgres, website-postgres

  These are off-the-shelf images (postgres, redis, the kyverno CLI, a search component) from public registries — exactly the "non-compliant registry" case the policy
  exists to stop. disallow-host-ports-check is armed but nothing violates it, so it produces zero gates.

  Tier 2 — Advisory warnings (17 Units, still appliable)

  Warnings don't block; they're the "tech-debt" tier. Distribution:

  - run-as-nonroot-enforcement-check — nearly universal (missing runAsNonRoot security context across the fleet).
  - liveness-and-readiness-probes-check — 14 Units lack probes. The three without this warning — eshop-api, portal-api, portal-frontend — already have probes.
  - resource-limits-enforcement-check — only 2: kyverno-cli-worker-unit and website-cms (the only multi-warning Unit, carrying all three).

  Tier 3 — Fully clean (2 Units, no gates, no warnings)

  aichat-api, aichat-frontend

  Want me to take any of this further — e.g. fix one gated Unit end-to-end to demo the release loop, or pull the actual kyverno failure messages for a specific Unit?

I asked the agent to look at the failures of one unit, and it proposed fixes.

Here are the full failure messages for docs-search (a apps/v1/StatefulSet, namespace currently the placeholder confighubplaceholder). It has 3 failing checks — 1
  blocking gate + 2 advisory warnings:

  🔴 Gate (blocks apply) — image-registry-restriction-check

  ▎ policy restrict-image-registries rule restrict-image-registries failed: Images must come from ghcr.io/acme/.

  The container image isn't in the ghcr.io/acme/ registry. This is the one that's actually refusing the apply.

  🟡 Warning (advisory) — liveness-and-readiness-probes-check

  ▎ policy require-probes rule validation failed: All containers must define a livenessProbe.

  🟡 Warning (advisory) — run-as-nonroot-enforcement-check

  ▎ policy require-run-as-nonroot rule validation failed: Pods or all containers must set runAsNonRoot to true.
  
  Notes

  - The resource name shows as confighubplaceholder/docs-search — the namespace is still a placeholder, so once you sort the registry you may also want ensure-namespaces /
  a real namespace before apply (a vet-placeholders gate would catch that too, if one were wired).
  - Fix loop to demo: set-container-image → ghcr.io/acme/docs-search:<tag> clears the gate; adding a livenessProbe and runAsNonRoot: true (e.g.
  set-pod-container-security-context-defaults, set-container-probe-defaults) clears the two warnings.

With changes made by functions, there’s no risk of typos, indentation issues, incorrect field names, etc., and functions can be applied to multiple resources in a single API call. Of course, the agent could also make changes directly. Claude Code likes to use yq, which it can invoke via set-yq without pulling all of the configuration into its context.

The configuration data in ConfigHub is shared state, so any changes can be observed immediately by other team members and agents, but they aren’t automatically released to the cluster. The agent can iterate until all checks pass.

Validating triggers provide the kind of guardrails that AI agents need in order to verify their work and guide their act-and-fix loops. I found them to be effective in forcing the AI agent to address gaps and inappropriate choices in configuration it generated with only high-level instructions. And much more quickly than waiting for CI.

Do you validate your Kubernetes configuration prior to deployment? What validation and/or policy tools do you use? Do you render and validate it only in CI? How long does it typically take to get a result? Does your AI agent understand how to find the results? Or do you have a way to render and run all the checks locally? How are the tools and checks packaged to ensure you’re running the correct, up-to-date set of checks?

Reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Kubernetes series.