I was working on ConfigMap updates in ConfigHub, and I bumped into a thorn in Kubernetes that I filed an issue for and tried to solve just over 10 years ago and is still unresolved in the project: ConfigMap rolling updates. What’s the problem, how did it happen, and how can you rollout ConfigMap changes?

(Side note: Notice that that issue number was well above 20,000 in less than two years after we open-sourced Kubernetes. GitHub sent email notifications for every comment on every issue and PR, and there were easily at least 10x as many of those. It was crazy.)

History

Kubernetes workload controllers, such as Deployment, StatefulSet, and DaemonSet, can automatically perform rolling updates across the Pods (groups of containers) owned by the workload controller when the podSpec is modified. This was a key feature of Deployment relative to ReplicationController (the precursor to ReplicaSet), for which rolling update was implemented in kubectl (and in the CLIs and scripts before that). (For more about the history of Kubernetes workload controllers, search for “Topic 4” in my 5-year anniversary history post.)

As I mentioned in my 10-year anniversary Road to 1.0 post, Deployment did not make it into Kubernetes 1.0. Deployment was inspired by an Openshift feature. Work on it was well underway in 1.1, but the automated changes made by both Deployment and HorizontalPodAutoscaler, which was also under development, destabilized the ReplicationController, which up to that point had just been modified by hand or by simple loops in scripts and the CLI. I worked on the ReplicationController/ReplicaSet controller and on the Deployment controller in order to get it ready to ship as beta in Kubernetes 1.2 — back in February 2016.

Releases 1.1 had taken 4-5 months and 1.2 was already taking more than 3 months. We were formalizing and automating the release process at the same time. For instance, Kubernetes 1.2 was the first release to produce a release-specific ChangeLog. There was not yet a formal concept of a release team, but I played the role of the release lead, as well as working on ReplicaSet and Deployment and wearing many other hats on the project, in CNCF, and on the GKE team. It was a busy time.

ConfigMap had also just been added as a non-secret Secret, essentially, to inject application configuration into containers. ConfigMaps could be referenced by Pods, and therefore by Deployments, to be injected as environment variables or volumes. Updating the podSpec of a Deployment would roll out new pods referencing the ConfigMaps specified in the podSpec template, which would update any environment variables or files projecting ConfigMap values, but updating the ConfigMap would…do nothing. At least at the workload controller level. With both Deployment and ConfigMap shipping at the same time, there wasn’t enough time to implement ConfigMap-specific rollout features.

Kubelet could update the ConfigMap volume in place, which could be detected by processes inside containers via inotify if watching. Most applications do not do that and rely on other approaches, such as reload on HUP, but if they did, they’d all update simultaneously unless there were some orchestration to control it. We had wanted to defer in-place rolling updates until we could update more properties, including resource limits, through Docker. Kubelet support also would be needed to restart containers to update environment variables set by ConfigMap values.

I did not want to add a configMapTemplates field to Deployment for rolling out ConfigMaps. ConfigMaps were already just data blobs in Etcd, and didn’t need to be replicated, though they would need to be versioned. Furthermore, adding them to Deployment would bloat that API and cause similar kinds of challenges for tools implementing ConfigMap support as embedding the PodSpec in multiple different resource types did. That type of embedding is what drove adoption of duck typing. I held out hope for referencing a separate PodTemplate API, which did exist at the time, and wanted to treat ConfigMap similarly. But that would introduce a new pattern, and it was hard to build consensus around an approach.

Cascading deletion and garbage collection were also just being implemented around that time, and shipped in release 1.3. The solution proposed was to copy the ConfigMap(s) along with the ReplicaSet and add an ownerReference to the ConfigMap referring to the ReplicaSet. There was a proposal this past December that was similar to that proposal from 2016. That approach could work, and I endorse it. Certainly there would have to be ownerReferences on generated ConfigMaps to ensure they would be cleaned up when the Deployment was deleted.

There were subsequent proposals in 2016, 2017, 2018, and 2019, and perhaps even after that, but none were accepted, unfortunately, partly due to various technical details, but largely due to non-technical factors, such as other priorities. For example, now the focus appears to be on AI workloads. There are also the challenges of stability and backward compatibility, especially now that the project is so widely used and the Deployment controller is such a core, critical component. Making and testing the changes would be a non-negligible amount of work, and maintainers would then be responsible for the code going forward.

The proposals had three main aspects:

  1. Triggering the pod rollout on a ConfigMap data change

  2. Copying ConfigMaps to version them similarly to ReplicaSets

  3. Garbage collecting or pruning old ConfigMap copies

Some of the proposals focused on a subset of these issues, such as triggering or garbage collection. And of course there were numerous other details, such as automatic vs. explicit triggering, making the new behavior opt-in for backward compatibility, dealing with shared ConfigMaps, whether to use the general garbage collector or to do the pruning in the Deployment controller along with ReplicaSet management, impact on garbage-collector load in the case of the former, support in StatefulSet and DaemonSet as well as Deployment, whether to add fields to the podSpec (which was shared between the workload controllers and Pods), whether to add fields to ConfigMap for the benefit of workload controllers, and so on.

Current solutions

Kustomize

As a workaround, we added the ability to Kustomize to generate ConfigMaps with unique names. Because Kustomize already needed to know which resource fields contained references to other resources in order to add name prefixes or suffixes, Kustomize could also update references to generated ConfigMaps. For example:

apiVersion: v1
kind: ConfigMap
metadata:
name: example-configmap-1-g4hk9g2ff8
...

apiVersion: apps/v1
kind: Deployment
name: my-app
...
volumes:
- configMap:
name: example-configmap-1-g4hk9g2ff8
name: config

Because the reference in the configMap.name field of the Deployment’s podSpec would be updated every time the data in the ConfigMap changed, that would trigger a rolling update by the Deployment controller. So kustomize did address (1) and (2), but not (3): keeping around old ConfigMaps that were still referenced, while deleting the ones that were not. So I don’t know how popular this approach is (about 61k search hits for configMapGenerator on GitHub), but only kustomize users would use it.

In general, pruning (i.e., deleting resources removed from the desired state) for kustomize is implemented by other tools, since kustomize doesn’t apply resources to the cluster itself. Pruning in kubectl apply identified applied resources using labels and was considered error prone, and an incorrect specification on the command line could cause resources to be deleted unintentionally.

Today, GitOps tools like ArgoCD and Flux handle pruning. For pruning to work properly with these tools, so that generated ConfigMaps would not be deleted while they were still in use during rolling updates or rollbacks, old revisions of the ConfigMaps would need to remain in the set of resources being applied until they were no longer needed by prior ReplicaSet revisions. I’m not aware of tools that do this kind of working-set management (except ConfigHub, which is discussed below).

Helm

Helm can generate ConfigMap contents of mutable ConfigMaps with stable names:

# Source - https://stackoverflow.com/a/65291822
# Posted by PjoterS
# Retrieved 2026-03-29, License - CC BY-SA 4.0

apiVersion: v1
kind: ConfigMap
metadata:
name: my-config
data:
{{- (.Files.Glob "configmap/*").AsConfig | nindent 2 }}

It can’t generate immutable ConfigMaps easily because Helm’s pruning mechanism would prune old revisions immediately when the ConfigMap’s name changed.

Helm has an “official” workaround for the rollout problem, which is to add a hash of a referenced ConfigMap to the podSpec template. This sets a pod annotation, but sometimes an environment variable value of a container is set instead:

kind: Deployment
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Like the kustomize resource name change, this also causes the podSpec template to be modified when the ConfigMap contents change. The key difference is that there’s only a single ConfigMap revision that is modified in place. This is mechanically simpler, but any pod that is created for any reason once the modified ConfigMap is applied will pick up the new ConfigMap data, and any processes using the inotify approach will pick up the new data immediately, which defeats the purpose of performing a rolling update.

Reloader

A number of tools have been developed by the community to address this gap, as well. A popular one, referenced in issue 22368, is Reloader. Reloader watches for changes and automatically triggers rollouts in workload controllers that are annotated:

apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
annotations:
reloader.stakater.com/auto: "true"
spec:
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: your-image
envFrom:
- configMapRef:
name: my-config
- secretRef:
name: my-secret

Like the Helm method, it also triggers a rolling update by modifying annotations in the podSpec template:

AnnotationTemplate: `{"spec":{"template":{"metadata":{"annotations":{"%s":"%s"}}}}}`,

While I wouldn’t say that approach is simple, it is simpler than copying ConfigMaps on write, updating references, and garbage collecting them when no longer needed. The ArgoCD project recommends this approach also.

ConfigHub

For ConfigHub, I implemented both approaches: the generation of unique, immutable ConfigMaps and simply updating the podSpec template with a hash of the ConfigMap.

In either case, ConfigHub generates the ConfigMap. Application configuration is stored in separate Units in its native format. Currently supported formats include YAML, JSON, Properties, INI, TOML, Env, and Text.

For example, Properties configuration might look like this:

database.port=5432
database.ssl.enabled=true

Storing the configuration in its native format makes it easier to understand, easier to validate, and easier to keep consistent across environments (e.g., with local development), and enables many standard ConfigHub functions to operate on it, since it is structured key/value data. It also works with ConfigHub’s variant-management mechanisms, for example enabling changes to be automatically merged across variants.

Unique, immutable ConfigMaps

To indicate to ConfigHub what name to use as the prefix of the name of the data key in the generated ConfigMap, a special property needs to be added to the configuration data:

configHub.configName=my-app-config

This property is omitted from the data when generating the ConfigMap. Using an Env file as an example:

apiVersion: v1
kind: ConfigMap
metadata:
name: my-config-03b362047a
...
immutable: true
data:
my-app-config.env: |
DATABASE_PORT=5433
DATABASE_SSL_ENABLED=false

The reference in the linked Deployment is updated via ConfigHub’s needs/provides value propagation mechanism. This is an example of a volume, but ConfigHub also supports envFrom:

volumes:
- name: config-volume
configMap:
name: my-config-03b362047a

ConfigHub knows where such reference fields are located in built-in Kubernetes APIs by using the list from kustomize. For non-built-in CustomResourceDefinitions, it’s possible to configure the locations in a couple different ways, but that’s a topic for another post.

By default, ConfigHub will automatically maintain the 10 most recently generated ConfigMaps, because that’s the default revisionHistoryLimit in Kubernetes. That number is configurable via a RevisionHistoryLimit option (see the documentation), which we could derive automatically from the Deployment in the future.

Mutable ConfigMaps

ConfigHub also supports mutable ConfigMaps that have stable names.

apiVersion: v1
kind: ConfigMap
metadata:
name: my-config

The Deployment references are similarly set automatically.

Additionally, ConfigHub will generate a hash of the ConfigMap and propagate it to an annotation in the podSpec template:

template:
metadata:
labels:
app: mydep
annotations:
confighub.com/Hash: a055c86133

That will trigger re-creation of the Deployment’s pods similarly to the Helm and Reloader solutions.

Wrap up

That hopefully helped you understand the two main alternatives for rolling out ConfigMap changes today:

  1. Generate unique, immutable ConfigMaps and update references to them. Delete old revisions of the ConfigMaps when they are no longer needed. This ensures correct, reliable rolling update behavior, even in cases where pods are re-created for other reasons during the rollout and where processes watch the file using inotify.

  2. Update the podSpec templates of workload controllers, such as Deployments, when the contents of referenced ConfigMaps (and/or Secrets) change, such as by generating and setting a hash value in an annotation. This is simpler and usually works well enough that the approach has been widely adopted.

I wouldn’t hold my breath for this being addressed in upstream Kubernetes at this point, but the feature is feasible. The change would have to be thoroughly tested, but I think it could be isolated to the Deployment controller.

Have you ever stumbled on this quirk of Kubernetes when making a change to a ConfigMap and wondering why your application didn’t observe the change? Did you comment on or +1 a comment on issue 22368? Do you generate ConfigMaps from native application configuration files with kustomize or with helm rather than maintaining them as ConfigMaps manually? Do you use Reloader, or do you use another tool?

Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Kubernetes series.