This topic came up as we, the ConfigHub cofounders, were speculating the future of operational tooling back in 2024. I didn’t find an article like this, so I decided to research the topic and relate the history to my own experience. References are at the end. I’m glad I waited to write this post, because the role of AI in operational tasks in the future, while not yet clear, has started to take shape.

I’m going to cover the generations of tools that have overlapped with my career. Like human generations, they overlap, but I’ll break them up approximately by decade. I have first-hand experience with some of the tools, but not with most of them. In some cases the tools were adjacent to what I was working on at the time, and Google developed its own internal tools. I’m most familiar with the Cloud Native era that arose around Kubernetes, Terraform, and the cloud.

There are clear recurring patterns: increased automation to cope with increased scale of machines and deployments, increased version-control complexity to cope with increased scale of collaboration, adoption of new technologies to save costs, waves of fragmentation and consolidation, and waves of services and self-hosting.

What do you think will come next?

This is kind of long since it covers four decades. If you aren’t interested in the history, you can skip to the AI section (search for “2023-today”) near the end.

Late 1980s: the batch-and-mainframe-to-Unix transition and ITIL

This era is dominated by tools that assumed a modest number of expensive, long-lived machines managed by hand. In this era, I used Apple computers, PCs, and Digital VAX minicomputers through attached terminals and dial-up modem network connections.

Version control was already a generation old. SCCS, developed in 1972, was nearly the first version control tool. RCS was released in 1982 at Purdue, where I went to college, as a free alternative. RCS was, naturally, the first version-control system I used. RCS was used as an ops tool as well as a dev tool. It was used to version Unix config files and scripts, not just source code. CVS was created in 1986 as a set of shell scripts layered on top of RCS to add repository-level tracking and a client-server model, but I didn’t use it in that form.

Batch scheduling / queuing in high-performance computing (HPC) traces to this period. The Network Queueing System (NQS) is considered the progenitor of essentially all later batch schedulers — it originated at NASA Ames around 1986. Condor (now HTCondor) started around 1988 at the University of Wisconsin, built around the idea of cycle-scavenging idle desktop workstations. Great name. I became exposed to this area at the beginning of the next decade, but both of these tools were influential.

Process/service management meant mainly init and cron . There was no "process supervisor" concept yet AFAIK, just rc scripts. Different systems used different distributions of Unix based on SysV Unix (IRIX, AIX) and BSD Unix (Vax, SunOS, Ultrix). Purdue had all of those systems, and more.

Process/ITSM frameworks were born at this time. The British government’s CCTA released the original ITIL in 1989, and the first ITIL books were Service Level Management, Help Desk (incorporating Incident Management), Contingency Planning, and Change Management. The CMDB concept came directly out of this framework — the ITIL principles from the late 1980s called for a database to track IT services, making the CMDB idea nearly as old as ITIL itself. On the commercial side, Remedy Corporation shipped its Action Request System in 1991, which became the dominant help-desk/ITSM ticketing engine for over a decade.

Early-to-mid 1990s: commodity Unix, Linux, and clusters

The defining shift was cheap commodity hardware and the arrival of Linux, which suddenly meant there were many machines to manage. During this decade I used PCs, minicomputers, workstations, and supercomputers.

CVS matured into the version-control tool of choice. It was rewritten in C, and CVS 1.0 was submitted to the Free Software Foundation in November 1990. CVS, which supported merging changes, definitely made it easier to collaborate on software projects compared to RCS, which locked files. I used both version-control systems over NFS, AFS, and other network filesystems rather than with a central hub service.

Configuration management tools were invented. CFEngine was created in 1993 to automate management of a group of workstations. Its lasting contributions were the ideas of declarative desired state and convergence (self-healing toward a defined state rather than running imperative scripts once) — the intellectual foundation for everything that followed. Mark Burgess’s 1995 USENIX paper “Cfengine: a site configuration engine” is the canonical primary source. I didn’t have the need to use it at the time.

Clustering captured the zeitgeist. The title of a conference talk/paper from Lawrence Livermore National Lab (LLNL) in 1990 was “Attack of the Killer Micros”. Beowulf originally referred to a specific machine built in 1994 at NASA — a 16-node cluster of Intel DX4 processors on channel-bonded Ethernet, built at CESDIS at Goddard under the HPCC/ESS project. The term “cluster of workstations” became prevalent. The concept, if not yet the reality, was commodity off-the-shelf (COTS) hardware plus open-source software replacing supercomputers. I interned at LLNL in the the early 90s, and NASA Ames was a 45-minute drive from there. I evaluated PVM, though mostly on distributed-memory supercomputers rather than workstations, and participated in the creation of MPI, particularly the collective communication model and libraries. I was a user rather than an operator of these systems.

Workload management was commercialized. Platform Computing was founded in 1992, and its LSF product was based on the Utopia research project at the University of Toronto. LSF became the standard commercial scheduler in finance and semiconductor HPC. In the mid-1990s NASA Ames developed the Portable Batch System (PBS) as an NQS replacement, spawning the OpenPBS → PBS Pro → TORQUE lineage. These queueing systems had similar CLI interfaces to submit jobs: bsub, qsub, etc. They were convenient for launching and managing long-running (multi-hour) tests, simulations, and batch jobs on shared pools of machines, even if just a handful of machines. Some supercomputers had similar interfaces, but for parallel workloads.

Package managers appeared. dpkg was created for Debian in January 1994, first as a shell script, later rewritten in Perl and then C. RPM was written in 1997 by Erik Troan and Marc Ewing, cofounder of Red Hat— though a primitive RPM shipped with Red Hat Linux 2.0 back in 1995. I interned with Marc at IBM in 1991. We built a system for distributing files to a network of AIX workstations because the local network, and network filesystem, went down frequently. I don’t know if that was part of the inspiration for RPM, but a package manager would have been useful.

Network monitoring got its first graphing tool. MRTG (Multi Router Traffic Grapher) was created in 1995 to monitor router traffic via SNMP, producing time series bandwidth graphs. Big Brother, one of the first tools to give a red/green web dashboard of host and service state, appeared around 1996–1997. The purpose wasn’t exactly the same, but at LLNL in the early 1990s I used ParaGraph and Pablo profiling/tracing tools to visualize supercomputer communication overheads over time, so similar ideas were emerging in multiple areas.

ssh was developed. In 1995. The network tools I had used up to that point — telnet, ftp, etc. — did not use encryption. We could telnet to SMTP servers to send email as anyone. Simpler times, if less secure. The Internet was just entering the public awareness with the emergence of the World Wide Web.

Late 1990s: monitoring, supervision, and dependency-aware packaging

I was in grad school focusing on compilers at the time rather than working at a Dot-Com, and not paying much attention to operational tools, so this period was filled in by searching. Let me know if I missed something important or got some details wrong.

Nagios arrived (as NetSaint). NetSaint was released in 1999, and renamed to Nagios in 2002. Its plugin architecture, a core scheduler running arbitrary check plugins, made it the default open-source monitoring system for over a decade, and it was the ancestor of Icinga, Shinken, and Naemon. I think I encountered Nagios, or at least heard of it, at Transmeta in the early 2000s.

RRDtool reshaped metrics storage. RRDtool was released in 1999 as a spin-off of MRTG, using a round-robin (fixed-size, circular-buffer) database so storage stayed constant over time. It became the storage/graphing backend for nearly every monitoring tool of the 2000s. Storage was a lot more limited and expensive then compared to now, so bounded storage was a necessity, unlike today’s scalable timeseries databases.

Process supervision became a named concept. daemontools (with svscan and supervise) established the modern pattern — a master process that stays parent to each service and restarts it immediately on crash, with reliable logging. It inspired runit and other process supervisors.

APT added dependency management. The Advanced Package Tool was developed for Debian in 1998, as a high-level front-end to dpkg that resolved dependencies and fetched from centralized repositories. yum brought the same capability to RPM systems a couple of years later. Dependency management is so essential now, that it’s hard to imagine that packages didn’t always provide it, though containers later repeated a similar pattern.

2000s: DevOps groundwork and the VM explosion

The combination of multi-core chips, hardware acceleration, and the Great Recession ushered in the virtual machine (VM) era, to save costs through consolidation, creating even more “machines” to manage. Steve Herrod was my manager at Transmeta for a week, before he left for VMWare. Containers in Linux (cgroups) were developed for similar reasons (i.e., binpacking) around the time I joined Google. Google skipped VMs and adopted containers on bare metal. Go was developed at Google late in the decade, also.

Version control split into centralized-vs-distributed. Subversion (SVN) was created in 2000 to be “CVS done right,” with atomic commits and efficient branching; version 1.0 shipped in February 2004. That was the main version-control tool I used immediately prior to joining Google. Meanwhile BitKeeper (a proprietary distributed VCS founded around 2000) was used by the Linux kernel from 2002 to 2005. When the free BitKeeper license was revoked in April 2005, Git and Mercurial were both created in response. Linus had already left Transmeta at that point to focus on Linux, so I wasn’t immediately aware of git. I used Mercurial a little, and the fully distributed model was challenging to keep sane in the absence of a central hub like Github. Google used Perforce, which had a centralized model.

Configuration management became mainstream. Puppet was released in 2005. Puppet adopted CFEngine’s declarative desired-state idea and added a Ruby-based DSL with a client-server model. Chef followed in 2009, and was the predominant alternative alongside Puppet and CFEngine, taking a more imperative Ruby "recipes" approach. CFEngine itself was rewritten as CFEngine 3 in 2008 around Promise Theory. GitHub was founded around the same time and found early traction in the Ruby community. Google had its own internal solution to this problem, at least for its datacenter machines, designed to manage its massive fleet of systems underlying Borg.

Monitoring fragmented. Cacti was started in 2001 as a friendlier PHP front-end to RRDtool, aiming for more ease of use than RRDtool and more flexibility than MRTG. Ganglia targeted large HPC clusters with a lightweight multicast agent. Zabbix was released in 2001 as an integrated monitoring platform, and Munin (~2003) offered RRD-based node graphing. The Nagios fork Icinga appeared in 2009. Within Google, the BorgMon monitoring tool was developed, which later served as the inspiration for Prometheus.

Init systems evolved. Ubuntu’s Upstart (Canonical, 2006) introduced event-based booting as a SysV-init successor, and runit (~2004) carried the daemontools approach forward.

Continuous integration emerged as tooling. CruiseControl, produced by ThoughtWorks, was the venerable early CI server (2001), preceded by Mozilla’s Tinderbox (~1998). Hudson was created in summer 2004 at Sun Microsystems and first released in February 2005. It overtook CruiseControl in popularity in 2008 and by 2010 held an estimated 70% of the CI market. We implemented continuous, or at least daily, build-and-test automation at Transmeta and PeakStream, but I don’t recall using a CI tool. I think it was just cron and a queuing system. Again, Google had its own “continuous build” mechanism, developed to work with its massive monorepo codebase.

Service management went SaaS. ITIL v2 (2000–2001) consolidated the framework into the well-known Service Support and Service Delivery volumes, and ITIL v3 in 2007 renamed the process to Service Asset and Configuration Management, cementing the CMDB’s central role. Self-hosted ITSM products like BMC Remedy and HP Service manager dominated in this decade, but ServiceNow was founded in 2004 and would go on to dominate ITSM/CMDB tools in the future.

Infrastructure as a Service. AWS launched in 2006, which was essentially the beginning of the public cloud.

Platform as a Service. Heroku launched in 2007, and AppEngine launched in 2008. CloudFoundry was conceived in 2009. By the early 2010s, there were many PaaS offerings, which were soon disrupted by Kubernetes.

Cloud Management Platforms. Rightscale was founded in 2007. The formula of a centralized management control plane plus policy enforcement and cost management was carried forward into both the infrastructure-as-code world (“TACOS”) and Kubernetes world.

2010–2013: consolidation into “DevOps”

Jenkins forked from Hudson. After Oracle acquired Sun and claimed the Hudson trademark, the community voted to rename the project Jenkins. The first Jenkins release shipped in 2011 under the MIT license, and roughly 85% of the community migrated within months. When Kubernetes started, Jenkins was still extremely popular, and the project adopted it, until the scale became too large to manage.

systemd rewrote Linux init. Introduced in 2010, systemd absorbed process supervision, socket activation, logging, timers, and mount management into PID 1 — controversially subsuming much of what daemontools, Upstart, and separate cron/inetd tooling had done. This was the last major process management change before the container era. Systemd was used together with Docker initially.

New configuration management tools, and YAML. Ansible arrived in 2012, with an agentless SSH-plus-YAML approach — a reaction to the agent-and-DSL complexity of Puppet and Chef. Ansible was growing rapidly in popularity around the time we started working on Kubernetes, and was commonly used to manage the nodes of Kubernetes clusters, even though we initially implemented node setup using Salt, another new config-management tool. Popular CI SaaS products, such as TravisCI and CircleCI, also adopted YAML. YAML became pervasive in the DevOps tools space.

Infrastructure as Code. AWS CloudFormation launched in 2010–2011. It later served as the inspiration for Terraform.

2013–2022: Cloud Native, Containers, Kubernetes, GitOps, X as Code, and the later ZIRP era

As the original lead architect of Kubernetes, Kubernetes Steering Committee member, and CNCF Technical Oversight Committee member, I was heavily involved in this era. I already wrote about the history of Kubernetes and about the CNCF landscape, so I won’t rehash those topics in this post.

Containers. Docker was first shown publicly March 2013. It bundled cgroups, namespaces, LXC, union filesystems, and other Linux primitives into a developer-friendly image format and runtime. That obviously had a huge impact on the operational landscape. The Open Container Initiative formed in June 2015 to standardize image and runtime formats, and containerd was contributed to the CNCF in 2017.

Container orchestration. An effort to build a container-based platform started within Google later in 2013. We open-sourced Kubernetes at the first DockerCon in June 2014. The Cloud Native Computing Foundation (CNCF) was kicked off in July 2015, coinciding with Kubernetes 1.0. Kubernetes was the leading “container orchestrator” by the end of 2017. Not only did the cloud Kubernetes services (GKE, EKS, etc.) become popular, but Kubernetes also was widely adopted “on prem”.

Whitebox monitoring and multi-faceted observability. Prometheus was developed in 2012, inspired by Google’s Borgmon, and was open-sourced in January 2015. It was the second project to join CNCF, in 2016. Its whitebox pull-based, label-based, dimensional metrics model became the de facto monitoring standard for Kubernetes workloads. OpenTelemetry, also a CNCF project, formed in 2019 by merging OpenTracing and Google’s OpenCensus into one vendor-neutral standard for metrics, logs, and traces, which was adopted by major observability vendors. Grafana, released in 2014, became popular for visualization. Timeseries databases were developed for metric storage. Elasticsearch became popular for searching logs.

Service mesh appeared. Linkerd (2016) coined the term “service mesh”. Service meshes initially used sidecar containers to proxy traffic. Envoy was open-sourced (2016), serving as a data plane. Istio launched in 2017 as a service-mesh control plane. All of these projects eventually landed in CNCF.

GitHub Actions disrupted the SaaS CI tool space in 2019.

Kubernetes configuration as code. Helm (2015–2016; discussed some in this post) became the de facto Kubernetes package manager and templating tool. Kustomize was created in 2017. Dozens upon dozens of other configuration generation tools were developed.

**GitOps was coined** by my cofounder, Alexis Richardson, in 2017 — Git as the source of truth, with a controller continuously reconciling actual state to the desired state. Weaveworks’ Flux (2016) and Intuit’s **Argo CD** (~2018), both now in CNCF, became the two most popular GitOps tools.

Infrastructure as Code was widely adopted, starting with Terraform (using HCL) in 2014, and then Pulumi (using general-purpose programming languages) and Crossplane (using KRM) in 2018. Terraform had the largest ecosystem of tools built around it. A lot of those tools shifted to OpenTofu later.

Configuration management (the OS/filesystem variety) faded. All of the major tools, Puppet, Chef, Ansible, and Salt, were acquired, and their popularity diminished with the rise in popularity of API-driven operations.

Policy as Code arrived. Open Policy Agent / OPA (2016) with Gatekeeper for admission control, later joined by Kyverno (2020).

**Secret management became essential**. Vault was very popular, at least until the license change to BSL, but there were many other tools.

**Asset management tools came to the cloud**. To help manage security and cost.

Platform engineering tools and practices developed mindshare. Kubernetes users built internal developer platforms almost from the beginning, but the concept really took off around 2018. Backstage was open-sourced in 2020 and donated to CNCF, seeding the developer-portal trend. Part of the motivation of both movements was to hide some of the complexity caused by the tool proliferation of the era.

ITSM partially connected with DevOps. Jira Service Desk (2013) integrated with Atlassian’s developer-oriented tools, like Bitbucket Pipelines and Confluence. ServiceNow acquired and developed some DevOps capabilities, also, but I’m unsure how they fared.

2023-today: Post-ZIRP and AI

Obviously AI isn’t the only recent development. eBPF is replacing service-mesh sidecar containers, for example. OpenTofu was forked from Terraform due to the BSL license change, which was part of a bigger trend of open-source license reversals. ClickHouse’s column storage is now used to store observability data.

It’s already clear that AI will define the current era, though.

Tools and services built to run AI workloads. Inference gateways, Modelplane, AICR, inference services, neoclouds, etc. Kubernetes was extended with AI-oriented features, such as DRA, also. That this area is getting attention is a no-brainer, but it is still in the early stages.

Git is under pressure. There have been shifts in version-control tools before. And similar to past responses to increases in scale, the large volume of changes made by AI agents is motivating challenges to git, or at least its current implementation, such as GitLab’s “act 2”.

Terminal-based AI agents have made it possible for AI to use current tools (especially CLIs), which the LLMs were also trained on. Due to the prevalence and increasing effectiveness of AI agents, agent experience is starting to supersede developer experience in importance. On the other hand, AI agents may also be able to replace some tools or at least some use cases, and can also be used to rapidly develop new tools. It’s still too early to guess how this will play out, but I’d guess that tools built by AI, for AI will win eventually. SWAMP is an example of that — AI-generated workflows for AI agents to execute, using an AI-authored tool.

New tools aren’t in the models’ training sets, however, so that’s where agent skills and/or custom harnesses, like Stakpak, need to fill the gap. Where CLIs aren’t available, MCP servers can provide access to tools, and agents can use browsers as well, though sometimes they get blocked. Tool selection, context management, memory, sharing information across individuals, and safety are current challenges.

A number of recent reddit threads indicate that many people are not ready to let agents make changes to production environments yet, which is a significant contrast with software development, where autonomous “loops” are a hot topic. Approaches I’ve seen to constrain AI agents include using credentials with read-only permission, MCP servers to evaluate CLI commands and other requests and/or make read-only API requests, secret proxies, network sandboxes with policy gates, deny-tools command lists, PreToolUse hooks that vet command patterns, and specialized harnesses with policy layers. The patterns to separate “safe” from “unsafe” changes appear coarse-grained so far. Presumably coding agents are being used to make changes to configuration as code in git, with human review, but I haven’t seen a lot of discussion about it yet, other than for creating new Helm charts and Kubernetes manifests. ConfigHub provides a governance layer that can validate changes automatically as well as gate changes on human review, and we are working on agent-based techniques as well.

Root-cause analysis (RCA), troubleshooting, incident response, and on-call agents are a current trend, collecting and interpreting multiple data sources quickly**:**

I’d imagine that this could drive significant changes to observability tools and other tools used during incident management (e.g., Slack, Jira).

Alternatives to Infrastructure/configuration as code. Formae is an IaC tool that can generate Pkl code from live infrastructure. System Initiative launched and was archived, in favor of SWAMP, which was mentioned earlier. Spacelift Intent directly calls provider APIs without using OpenTofu or other IaC tool. I’ve seen at least one example of someone replacing Helm with markdown files also. With ConfigHub, we’re using the approach of using AI to generate tools to manipulate configuration as data.

Infrastructure/configuration as code tools and formats were developed by software engineers for (human) software engineers. Human software-engineering instincts are already strained by IaC/CaC. AI agents don’t have the same constraints and preferences or strengths and weaknesses, and design choices made for existing tools may not be the best choices for agents — DX vs. AX.

I think the question is not whether “X as code”, at least as we know it, will be superseded, but what will it be replaced by.

What operational tools do you remember using earlier in your career? How do you think AI will shape future tools? Are you already using AI in operations? In production? To make changes? What safety mechanisms do you use? Have you used AI to build any new operational tools, even if just for yourself?

Feel free to reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Infrastructure as Code and Declarative Configuration series.

References

Version control

Configuration management

Clustering / HPC

Batch schedulers / queuing

Monitoring / observability

CI

Package managers

Process / service supervision

ITSM / ITIL / CMDB

DevOps