I wanted to run vLLM on Kubernetes. There were several options:
While there was a little documentation for the Operator, the documentation focused on the Helm chart, and most features (e.g., prefix-aware routing) only had examples using the Helm chart. The Claude/agent skill and plain YAML were quite simple, and didn’t explicitly support most features, which were presumably covered elsewhere in the documentation. I liked the idea of using an agent skill for installation, though.
As I’ve mentioned before, Helm charts that act as installers for off-the-shelf applications can be quite complicated because they frequently enable significant customization of the configuration. Rather than trying to figure out how to use the Helm chart myself, I used Claude Code.
First I asked Claude what the Operator could do that the Helm chart couldn’t do, and the answer was nothing really. On the other hand, the Helm chart supported many more features: Ray, autoscaling, Prometheus, and more. So, I did need to start with the Helm chart.
The Helm chart had a values.schema.json file, but it did not seem complete. I asked Claude for an analysis:
The helm chart's values.schema.json file is significantly incomplete. It covers only a fraction of the supported values.
What the schema covers
Only 3 top-level properties: servingEngineSpec, routerSpec, and cacheserverSpec — and even within those, coverage is partial. The schema handles the basics of modelSpec[] items reasonably well (name,
repository, tag, resources, GPU, PVC, probes, KEDA).
What's missing from the schema
6 entire top-level sections:
loraAdapters, loraController, extraObjects, grafanaDashboards, kube-prometheus-stack, prometheus-adapter
~10 servingEngineSpec-level properties, including:
enableEngine, configs, strategy, readinessProbe, sidecar, serviceMonitor, imagePullPolicy, env
~15+ modelSpec[] item properties, including:
chatTemplate, enableLoRA, enableTool, toolCallParser, shmSize, extraVolumes, extraVolumeMounts, affinity, raySpec, envFromSecret, storageClass, pvcAccessMode
~20+ lmcacheConfig sub-properties (all NIXL, PD, controller, and p2p fields)
~7+ vllmConfig sub-properties (tensorParallelSize, maxNumSeqs, maxLoras, gpuMemoryUtilization, v0, v1, prometheusMultiprocDir)
~15+ routerSpec properties (enableRouter, repository, tag, resources, serviceType, ingress, route, affinity, readinessProbe, serviceMonitor, etc.)
There are also a few values referenced in templates that aren't documented in values.yaml either (e.g., sharedPvcStorage, routerSpec.hf_token, pvcStorage as an object with emptyDir).
As is fairly common, the values.yaml file used comments for documentation. For example:
# -- LoRA Adapter Configuration
loraAdapters: []
# -- LoRA adapter instances to deploy
# Each instance should contain:
# - name: (string) The name of the LoRA adapter instance
# - baseModel: (string) The name of the base model this adapter is for
# - vllmApiKey: (optional) API key configuration for vLLM authentication
# - secretRef: Reference to a secret containing the API key
# - secretName: (string) Name of the secret
# - secretKey: (string) Key in the secret containing the API key
# - value: (string) Direct API key value
# - adapterSource: (object) Configuration for the adapter source
# - type: (string) Type of adapter source (local, s3, http, huggingface)
# - adapterName: (string) Name of the adapter to apply
# - adapterPath: (optional, string) Path to the LoRA adapter weights
# - repository: (optional, string) Repository to get the LoRA adapter from
# - pattern: (optional, string) Pattern to use for the adapter name
# - maxAdapters: (optional, int) Maximum number of adapters to load
# - credentials: (optional, object) Reference to secret with storage credentials
# - secretName: (string) Name of the secret
# - secretKey: (string) Key in the secret containing the credentials
# - loraAdapterDeploymentConfig: (object) Configuration for adapter deployment
# - algorithm: (string) Placement algorithm to use (default, ordered, equalized)
# - replicas: (optional, int) Number of replicas that should load this adapter
# - labels: (optional, map) Additional labels for the LoRA adapter
#
# Example:
# loraAdapters:
# - name: "llama3-nemoguard-adapter"
# baseModel: "llama3-8b-instr"
# vllmApiKey:
# secretRef:
# secretName: "vllm-api-key"
# secretKey: "VLLM_API_KEY"
# adapterSource:
# type: "local"
# adapterName: "llama-3.1-nemoguard-8b-topic-control"
# adapterPath: "/data/lora-adapters/llama-3.1-nemoguard-8b-topic-control"
# loraAdapterDeploymentConfig:
# algorithm: "default"
# replicas: 1
# labels:
# environment: "production"
# model: "llama3-nemoguard"
The loraAdapters configuration above is used almost verbatim in a CRD:
{{- if .Values.loraAdapters }}
{{- range .Values.loraAdapters }}
---
apiVersion: production-stack.vllm.ai/v1alpha1
kind: LoraAdapter
metadata:
name: {{ .name }}
namespace: {{ $.Release.Namespace }}
labels:
app.kubernetes.io/component: lora-adapter
helm-release-name: {{ $.Release.Name }}
spec:
baseModel: {{ .baseModel | quote }}
{{- if and (not .vllmApiKey) ($.Values.servingEngineSpec.vllmApiKey) }}
vllmApiKey:
secretName: {{ $.Release.Name }}-secrets
secretKey: vllmApiKey
{{- else if .vllmApiKey }}
vllmApiKey:
secretName: {{ .vllmApiKey.secretName | quote }}
secretKey: {{ .vllmApiKey.secretKey | quote }}
{{- end }}
adapterSource:
type: {{ .adapterSource.type | quote }}
adapterName: {{ .adapterSource.adapterName | quote }}
{{- if .adapterSource.adapterPath }}
adapterPath: {{ .adapterSource.adapterPath | quote }}
{{- end }}
{{- if .adapterSource.repository }}
repository: {{ .adapterSource.repository | quote }}
{{- end }}
{{- if .adapterSource.pattern }}
pattern: {{ .adapterSource.pattern | quote }}
{{- end }}
{{- if .adapterSource.maxAdapters }}
maxAdapters: {{ .adapterSource.maxAdapters }}
{{- end }}
{{- if .adapterSource.credentials }}
{{- if (kindIs "string" .adapterSource.credentials) }}
credentialsSecretRef:
name: {{ $.Release.Name }}-secrets
key: lora_adapter_credentials_{{ .adapterSource.adapterName }}
{{- else }}
credentialsSecretRef:
name: {{ .adapterSource.credentials.secretName | quote }}
key: {{ .adapterSource.credentials.secretKey | quote }}
{{- end }}
{{- end }}
loraAdapterDeploymentConfig:
algorithm: {{ .loraAdapterDeploymentConfig.algorithm | default "default" | quote }}
{{- if .loraAdapterDeploymentConfig.replicas }}
replicas: {{ .loraAdapterDeploymentConfig.replicas }}
{{- end }}
{{- end }}
{{- end }}
Given the largely 1:1 value-to-field correspondence, and the use of examples both in the documentation and in the values.yaml file, I wondered what the point was of using Helm templates at all. I wanted to cut out that whole layer of unnecessary complexity. (This issue isn’t Helm-specific. It also applies to domain-specific languages for configuration generation and general-purpose-language-based generators.)
So, I asked Claude to create a Claude/agent skill to install vLLM:
I want to create an agent skill that is as comprehensive as the helm chart, but that does not use helm, kustomize, or the Operator. I just want to use plain Kubernetes YAML files and select and modify them as necessary. The skill should be able to make use of the vLLM features supported by the helm chart.
I didn’t want to directly install vLLM, however. I wanted to manage vLLM through ConfigHub:
Instead of deploying the resources directly to a Kubernetes cluster, I want to create units in ConfigHub using its CLI, cub.
That didn’t quite work in one shot. We need to develop a skill to teach AI agents how to use ConfigHub. But using our documentation and --help mostly worked fine. We previously experimented with MCP, but didn’t find it to work better than our CLI.
SKILL.md (429 lines) — The skill definition with:
- Trigger: activates when the user wants to deploy vLLM on Kubernetes via ConfigHub
- 6-step deployment procedure: gather requirements, create space, prepare/create units, set namespaces, verify, report summary
- Comprehensive configuration reference for all vLLM features
- Instructions to use cub space create and cub unit create for each resource
19 template YAML files covering all components from the helm chart:
┌────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Category │ Templates │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Engine (per model) │ engine-deployment.yaml, engine-service.yaml, engine-pvc.yaml, engine-pdb.yaml, engine-configmap.yaml │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Router │ router-deployment.yaml, router-service.yaml, router-serviceaccount.yaml, router-role.yaml, router-rolebinding.yaml, router-hpa.yaml, router-pdb.yaml, router-ingress.yaml │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Cache Server │ cache-server-deployment.yaml, cache-server-service.yaml │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Autoscaling │ engine-keda-scaledobject.yaml │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Monitoring │ engine-servicemonitor.yaml, router-servicemonitor.yaml │
├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Secrets │ secrets.yaml │
└────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Key design decisions:
- Each template is a single Kubernetes resource (1:1 mapping to ConfigHub units)
- Uses confighubplaceholder for values requiring user input (model URL, etc.) and MODELNAME as a per-model substitution token
- Templates use reasonable defaults matching the helm chart (ports, probes, labels)
- The SKILL.md documents how to add every optional feature (LMCache, tensor parallelism, LoRA, NIXL, chat templates, init containers, etc.) as incremental YAML modifications rather than Helm
conditionals
The skill required just a few iterations. I put it here:
https://github.com/bgrant0607/vllm-skill
I didn’t spend too long on this, so I’m sure it could be improved. I also haven’t tested it much yet. It’s interesting that Claude replaces example strings in commands directly, but used low-level tools like yq to perform substitutions in configuration. I’ll need to experiment with that more.
However, there are already some clear benefits to using Claude skills (or other agent skills) over using Helm to install the application.
First, it’s interactive. Claude asks about whether I want to use specific optional features:
-
Do you need a request router? (default: yes, if multiple models or production use)
-
Do you need persistent storage for model weights? (default: no)
-
Do you need a Hugging Face token for gated models?
-
Do you need autoscaling (KEDA for engines, HPA for router)?
-
Do you need monitoring (ServiceMonitor)?
-
Do you need an Ingress for external access?
-
Do you need an LMCache server?
-
Routing strategy: roundrobin (default), session, prefixaware, or kvaware?
Second, it’s high-level. I don’t need to specify every Helm value individually.
Third, both the “template” configuration and the final configuration are both human readable and can be edited by tools, including ConfigHub functions. For instance, this is the PVC configuration:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: vllm-MODELNAME-storage-claim
namespace: confighubplaceholder
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
Fourth, the final configuration can be immediately validated by a number of linting, validation, and policy tools. This can be done using Triggers in ConfigHub.
Fifth, I can make subsequent changes using any method I prefer, which might be using the skill, or could just be ConfigHub functions for common Kubernetes configuration changes, like changing resource requests (there’s a set-container-resources function). When I know what Kubernetes resource field I need to change, I don’t need to figure out what the corresponding Helm value is. Configuration can be created using any method you like, and then edits can be made using any method you like. They aren’t coupled. You can use the best tool for the task and situation.
More generally, AI agents are pretty good at generating and editing Kubernetes workload configuration at this point, especially plain YAML manifests and standard resource types like Deployment and Service. Rather than trying to factor out parts of the generated configuration that need to be customized for different uses or environments using Helm, kustomize, cdk8s, or other configuration generation tool, to make the configuration “DRY”, which would make the configuration more complicated, the combination of AI and ConfigHub make it viable to generate and manage many copies (variants) of a configuration.
When configuration is represented and managed as data and can be read and written via API, configuration can be managed the way that other data in databases is managed, thereby avoiding the fundamental challenges with “as code” representations.
Have you used agent skills to customize Kubernetes configuration? Has your experience been positive or negative? Was it reliable enough? How difficult were the skills to use? How difficult were they to create? Do you have tips for how to make such skills more effective? Do you think skills or something else AI-based will replace Helm for installing off-the-shelf Kubernetes applications?
Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.
If you found this interesting, you may be interested in other posts in my Kubernetes series.
You are also welcome to try out ConfigHub, which is now in preview.