And why (and when) would anyone want that?
With GitOps as currently defined and implemented, a GitOps Operator will continuously apply the specified “desired” state from storage (git, OCI repositories, etc.).
That’s useful in the case that a single apply would fail whereas an indeterminate number of applies would succeed. Such non-persistent failures could occur due to a number of reasons: unexpressed dependencies, lack of strong consistency, broken network connections, process restarts, etc. A continuous retry loop, essentially.
And when the configuration is updated in storage, the live state should be updated accordingly.
The main reason for the continuous reconciliation loop, however, is continuous drift detection and remediation. That is, if the live state is found to not match the configuration in storage, the live state will be updated to match the state from storage.
Configuration drift happens for a variety of reasons. A common reason is that someone deliberately changed the live state. Why would someone do that when the configuration is specified in storage?
It could be that the person making the change wasn’t aware the configuration was managed in storage, though with Kubernetes configuration this is relatively unlikely, since managing configuration with version control has been a standard practice from the beginning.
It could be that someone was attempting to circumvent policy controls. Kubernetes RBAC and dynamic admission control can be used to prevent that.
It could be an accident, such as a command invoked on the wrong cluster or namespace.
A more common occurrence is that someone made a change out of necessity.
For instance, let’s say something breaks due to an application update, an unexpected operational issue such as a recurring OOM occurs , or an application needs to be updated urgently to address a security concern.
A number of people have told me that they believe it’s not only simpler and faster but even safer to “break glass” and just change the cluster directly.
If they use a GitOps tool, such as FluxCD or ArgoCD, they may need to suspend or disable it first. If they don’t, their change could be clobbered quickly, whatever problem they were trying to address.
I experienced something similar to that many years ago. I was on call. To address a production outage I needed to deploy a fix quickly. The normal process took a long time, too long. So I needed to do it another way. I deployed a new release and it fixed the problem. Seconds later, the previous release was redeployed by a reconciliation loop, taking the service down again. Disabling the reconciliation mechanism would take as long as using it to deploy, so I blocked its network access. Which would eventually be undone by another reconciliation loop.
Anyway, in the case that the operational change is successful and it needs to be made permanent, then it needs to be backported to the configuration source and the reconciliation mechanism needs to be re-enabled or unblocked. Typically, the configuration is in some kind of DRY (Don’t Repeat Yourself) format — templates, patches, etc. — so backporting the change is not trivial, at least not straightforward enough for a traditional deterministic tool to do. The values that need to be changed may be in a template, input variable values, value overrides, patches, configuration generation code, or the like. In the OOM case, we’d need to change container resources, as in this Helm chart excerpt:
resources:
{{- toYaml .Values.paymentService.resources | nindent 10 }}
In that case, the appropriate values file would need to be updated. If the values that needed to be changed weren’t parameterized, then the template would need to be changed, which would potentially affect all environments.
Both the configuration tool and the GitOps tool feel like they are getting in the way in this scenario. It should be straightforward to make these kinds of operational changes. You shouldn’t have to fight the automation.
ConfigHub maintains a one-to-one correspondence between the configuration in ConfigHub and the live state. If you do need to update the live state directly, the changes can be pulled back into ConfigHub automatically.
cub unit refresh --space apptique-prod paymentservice
When it is not desirable to keep the changes, they can be undone, just like any other changes.
cub unit update --patch --restore=-1 --space apptique-prod paymentservice
cub unit apply --space apptique-prod paymentservice
Synchronizing the live state back into ConfigHub provides more visibility into what the current live state is and what changes have been made. I like to see what the actual state is before making changes to the configuration, not just when I am about to apply the changes.
Now imagine continuously synchronizing the live state back into storage. That would be essentially “reverse GitOps”. However, we know when an apply operation is being performed, so we can synchronize in that direction as needed as well — Bidirectional GitOps.
That kind of bidirectional synchronization provides more control over operational changes than the current unidirectional approach, making deliberate operational changes less challenging.
That said, with the 1:1 correspondence and configuration represented as data, this kind of change is also safer (due to minimal blast radius) and much more straightforward to make through ConfigHub, reducing the need to update the live state directly. For common cases, such as changing resources, there are already pre-built functions for making such changes.
cub function invoke --space apptique-prod --unit paymentservice \
set-container-resources server floor 100m 200Mi 2
cub unit apply --space apptique-prod paymentservice
But you have the option to choose the approach that’s best in the situation when some urgent operational change is necessary.
What do you think? Have you needed to make operational changes and GitOps made the process more complicated and/or take longer? Does bidirectional GitOps sound promising, or confusing? Do you think digital twin is a better term to describe this pattern?
Feel free to email us at hello@confighub.com, or send me a message on LinkedIn, X/Twitter, or Bluesky.
You could also try out ConfigHub, which is now in preview.
If you found this interesting, you may be interested in other posts in my Infrastructure as Code and Declarative Configuration series or in my Kubernetes series.