DevOps: Nobody intends to build a wall of confusion

DevOps? Of course – we've had a Jenkins pipeline for testing our code for a long time. Just speak to Tobias, the head of the development team, or Annika from the Operations team about it. Right? – Well... not quite.

We encounter these or similar statements in our daily work time and again. In this two-part blog series, we want to present our view on the topic of DevOps and provide ideas on how you might be able to break through certain thought patterns or resolve misunderstandings.

Was ist DevOps?

The term DevOps is derived from the combination of the disciplines „Development“ and „Operations“. „Development“ generally refers to the creation of a system or software component. Many initially think of a larger software project, driven by multiple developers, addressing a significant problem area, having a REST API, or communicating with a database. However, even a 10-line Bash script that checks a database backup or automates the cleanup of test environment resources needs to be „developed“.

The term „operations“ describes the discipline of putting self-developed or externally provided software into service and ensuring that this system does not fall apart. As we hear time and again, this area is generally considered a necessary evil. After all, who fancies performing the monthly database restore test while others get to develop the next coolest feature for the software.

The cycle

But then: two of the team members heroically volunteer to take care of running the infrastructure and the developed software – the operations team is born.

Great, so now we have a Dev and an Ops team. Let's take a look at what the teams have to do based on the DevOps cycle.

The dev team takes care of the left-hand side of the loop. They plan what to do next, develop the software as they see fit, write documentation, build the APT package and the Docker container, and run their script for testing.

Working? Working! Version number 2.26.4 attached, packages released and briefly called the Ops team on the right to say they can now deploy the next version.

After the Ops team has quickly deleted that one faulty entry from the database, we're good to go. They'll grab the new version and their Python script to roll everything out to production. The database migration has run, the configuration has been adjusted as described in the documentation from the Dev team – Light, action, monitoring – rolling? … it's rolling, right? – Erm… no, the service keeps restarting every few minutes and, besides, there are now five new erroneous database entries.

Well, that happens. Actually, that happens all day, and essentially that's what DevOps is all about.

Interface == Vulnerability

In this two-team setup, there are at least two vulnerabilities, namely whenever one team has to hand things over to the other team.

In the first instance, it is the dev team who must describe in detail what they have developed or changed, what their thinking was behind it, and what specific considerations need to be taken into account during commissioning.

The second stage is the handover of results or problems that arose during the last commissioning in the last cycle. The Ops team would need to provide a report of all defects they noticed back to the Dev team's planning phase. Based on these findings, the next important steps could then be planned and developed to ensure the software's operation.

We only observe this step very rarely in practice. What is much more likely is that problems are not addressed at all. Reasons for this can be that the problems cannot yet be formulated very specifically, or because the Ops team does not want to expose themselves, perhaps having done something „wrong“ or not yet understanding the entire construct.

In the worst-case scenario, the people on the dev team are those who aren't developing the right things, and the ops team are those who lack sufficient knowledge of their „own product“ or „own service“. This is known as the “wall of confusion”. Both teams try to do what they believe is “right“ in their eyes, but they actually have no idea what's going on on the other side of the wall with the other team. The situation is also accompanied by an often unspoken conflict of interest. While the dev team constantly wants to push newly developed features into production, the ops team actually just wants to ensure stable operation of the software, which becomes more difficult with every new version or every change to the setup.

For this setup to work, a substantial, transparent, and honest transfer of knowledge must take place in both directions. This typically leads to two bottlenecks in said DevOps cycle, thereby increasing the duration from planning to the successful commissioning of a fix or feature. According to the “Theory of Constraints,“ the process chain of the DevOps cycle can only be traversed as quickly as the slowest or most inefficient step allows.

The dev team in the fast lane

It becomes problematic when there is no proper release management at all and changes are introduced into operation in a half-hearted manner. If the cycle repeats itself without the Ops team having fully caught up with the deployment, the workload on the Ops team continues to accumulate, while the Dev team injects new features with high motivation. This imbalance can then lead to the gap between the developed and the operated version becoming ever wider. Development work then takes place that, in the worst case, does not make it into production before the feature perhaps even becomes obsolete again.

So, what can we do to rectify this situation? This is described in Part 2.