If you’ve used Kubernetes kubectl apply, server-side apply, or kustomize, then you may have encountered the “strategic merge patch” feature. “Strategic merge patch” is a mouthful. What does it mean? In what sense is it “strategic”? Why does it exist?

I explained this several years ago and copied it into my first blog post, on the Technical History of Kubernetes. It’s also explained in the Kubernetes documentation, in the original design document, on Stackoverflow, and by some others’ blog posts. However, I was working on a new implementation for ConfigHub that could be used for non-Kubernetes configuration as well, and it seemed like a good topic to cover again, because the reasons why strategic merge patch exists may not be obvious.

Strategic merge patch was developed originally by Sam Ghods for use in kubectl apply, but it was necessary as a consequence of several other design decisions.

First, relatively early in the Kubernetes project, a decision had been made that map keys in the API should not have structural significance in the API. The main rationale was the difficulty in distinguishing map keys from fields in the API without a schema. Additionally, even with a JSON or OpenAPI schema, documentation of map keys must be embedded in the description of the whole map.

The original example was the specification of a port’s name using a field:

ports:
- name: www
hostPort: 80
containerPort: 80
protocol: tcp

Rather than this specification of its name using a map key:

ports:
www:
hostPort:80
containerPort:80
protocol: tcp

Kubernetes used the former approach for its API, where as configuration tools, such as the precursor to Docker Compose (Fig) chose the latter approach. The issue came up multiple times because common configuration tasks were made more difficult by this decision. Later, cases were added where order matters, however, such as initContainers, though that could have been solved using another field instead.

Second, I wanted the serialized Kubernetes API representation to be used as the basis for configuration formats, rather than developing and maintaining a parallel but slightly different representation for configuration.

Third, I wanted the configuration primitives to be tolerant of fields being set dynamically by automation. Borg’s configuration tool had ad hoc special-case logic to deal with such cases on a one-by-one basis, and the Borg API was modified to handle the most common cases, such as horizontal autoscaling. This was not an extensible approach and new automation components could be blocked until changes could be made to critical core libraries, since otherwise changes made by those components would be detected as configuration drift and would be undone by configuration tools.

That principle also enabled us to explicitly populate default values in the API, so that client tools would not need to guess how the API interpreted unspecified fields or read them from a schema. Also, a hardcoded schema would not work for fields set by admission control, which we felt was important to support organization-specific policies.

Fourth, while admission control and defaulting are synchronous, controllers, such as the horizontal pod autoscaler, operate asynchronously. That meant that if the state were completely overwritten without incorporating dynamically set fields, there would be a window of time where those API fields would be incorrect, and other controllers, such as the Deployment and ReplicaSet controllers, could act on those incorrect values.

The requirement to handle API fields being set asynchronously effectively necessitated merging of the desired state specified by configuration with the state in the API, rather than just replacing it completely. It turned out that merging was a key capability for configuration tools more generally, because declarative configuration creates copies of documents that need to be kept in sync.

Most configuration tools, such as Helm, generate configuration using a unidirectional process. When either the generation code (or templates) or input values change, the whole configuration must be regenerated. Such a generated configuration contains specifications for whole resources, but only statically specified values and does not include dynamically set values.

Other client tools, such as CLIs, can generally update selected fields of resources rather than just replacing whole resources. They typically use patch APIs to do that.

There are two standard JSON patch specifications, JSON Patch and JSON Merge Patch. JSON Patch specifies a sequence of imperative field changes: add, replace, remove, move, copy, test. JSON Merge Patch accepts a document (in this case a resource) using the same structure as the original. For configuration use cases, JSON Merge Patch appeared more convenient than JSON Patch.

Also, on the surface, a merge patch looked like it could be produced without diffing. However, this was not the case. Sometimes fields need to be removed. One example in Kubernetes is a change from one volume source, like emptyDir, to another, such as persistentVolumeClaim. Since they support different options, they are structured in the API as separate fields — a union/oneOf essentially. Identifying such a case automatically requires a diff with the previously specified desired state and the field to be removed must be assigned null in JSON Merge Patch.

To simplify this procedure for clients, I had proposed API support for automatically merging a resource specified by the client with fields stored in the server, by storing multiple sets of values. This proposal was not implemented, but API support for field tracking was eventually added years later in the form of server-side apply. Instead, to implement kubectl apply, we stored the previously specified desired state in an annotation. In fact, that was one of the original motivations for adding annotations to the API.

That solved the problem of diffing, but did not solve the problem of patching. The problem with JSON Merge Patch is its behavior with respect to arrays. From the RFC: “it is not possible to patch part of a target that is not an object, such as to replace just some of the values in an array.” That is a non-starter for the Kubernetes API, which extensively uses arrays, such as for the lists of containers in a pod, lists of ports as shown above, lists of environment variables, and so on.

So we needed to create our own patching mechanism in order to merge fields of array elements. But we also needed to handle the prohibition of the use of map keys. We can use the example of ports again. Let’s say a new port is inserted and another’s value is changed, and we want to merge with values in the API.

The original port:

ports:
- name: www
containerPort: 80
protocol: tcp

And let’s say a hostPort field was added to bind the www port to port 80 via a separate API call:

ports:
- name: www
containerPort: 80
hostPort: 80
protocol: tcp

And then a user changed the configuration to:

ports:
- name: observability
containerPort: 8090
protocol: tcp
- name: www
containerPort: 8080
protocol: tcp

If just merging elements of the array positionally, index by index, then that would result in this in the API:

ports:
- name: observability
containerPort: 8090
hostPort: 80
protocol: tcp
- name: www
containerPort: 8080
protocol: tcp

Rather than what was intended:

ports:
- name: observability
containerPort: 8090
protocol: tcp
- name: www
containerPort: 8080
hostPort: 80
protocol: tcp

This example may seem contrived, but the API has many such arrays, and it seemed likely that something like this would happen.

Essentially such arrays should be treated similarly to maps even though they aren’t structured as maps. I call them associative arrays.

That brings us back to strategic merge patch. “Strategic” refers to the ability to specify a patchStrategy for arrays, replace or merge. Additionally, in the case of merge, a patchMergeKey can be specified to identify the field of each element to use as the equivalent of a map key so that elements can be matched irrespective of their array positions.

To address the ports example above, the patchStrategy would be merge and the patchMergeKey would be name, to specify that the name field’s value should be used as the key for matching elements. That would enable the intended result to be achieved. However, in actuality, ports do not use the name field for merging, because the name field is optional. Instead, the containerPort field itself is used, along with the protocol, since TCP is not the only supported protocol. This can be seen in the Kubernetes API definition:

// +optional
// +patchMergeKey=containerPort
// +patchStrategy=merge
// +listType=map
// +listMapKey=containerPort
// +listMapKey=protocol
Ports []ContainerPort `json:"ports,omitempty" patchStrategy:"merge" patchMergeKey:"containerPort" protobuf:"bytes,6,rep,name=ports"`

But the mechanism does achieve the intended result for other cases, such as containers.

Similar behavior can be specified in CustomResourceDefinition using x-kubernetes-list-type map and x-kubernetes-list-map-keys.

We later reused strategic merge patch in Kustomize, to enable patching of what we called “overlays”, which are partial resource specifications. For example (from the documentation), to patch the spec.replicas field:

apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3

Kustomize gets information about array merge strategies and merge keys from OpenAPI documents.

Kustomize’s ability to patch configuration made it useful not just for patching hand-written Kubernetes YAML files, to create variants of the same configuration, but also to patch Kubernetes resources generated by other tools, such as Helm, enabling overrides.

ConfigHub supports managing variants by automatically merging changes from upstream configurations into downstream configurations, or, really, from any configuration into any other configuration, such as downstream to upstream or variant to variant.

As with Kustomize, ConfigHub can also use this mechanism to override values in generated configurations, also known as rendered manifests. Additionally, ConfigHub can merge changes to the live state back into ConfigHub, such as after a “break glass” operational change. This enables ConfigHub to maintain an accurate, up-to-date record of the state and eliminates artificial configuration drift.

Conceptually, the mechanism works similarly to Kustomize, but without the need to write patches by hand. It does that by computing a diff and encoding it in a form that can be used as a patch. To produce the patch correctly, the diff mechanism needs to be aware of array patch strategies and merge keys. Instead of requiring knowledge of the merge keys during both the diff and the patch processes, I chose to encode the merge keys in the patch.

More specifically, I encoded merge keys in resource field paths, as they would be in the case of maps rather than arrays. That was a convenient approach because a number of ConfigHub functions operate similarly in some ways to Kustomize transformers, and operate on specific resource fields. It can be more convenient for users of such functions to locate fields using merge keys, such as environment variable names, rather than array indices. It’s effectively a simple kind of search within the configuration. I’ll write another post in the future that goes into more detail on how these “visitor” functions work.

Anyway, that was a fairly deep dive on the history of strategic merge patch. It was implemented to support the merging of configuration into the live state in Kubernetes, but merging, in general, is a key configuration capability that it facilitates, and one that we support in a general-purpose way in ConfigHub.

Have you wondered why “strategic merge patch” is called that? Have you wondered why it’s needed? Do you wish it weren’t needed? Or have you wondered why other tools don’t support it? Have you experienced surprising or unfortunate merge behavior? Do you wish that arrays of command-line flags could be merged? Do CRDs that you use specify merge keys appropriately?

Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Kubernetes series.