Simplify! – Part 2

In the previous post, I described two observations:

1. IT has become an indispensable part of everyday business and private life.
2. Complexity on the IT solution side grows all the time approaching the verge of unmaintainability.

In this post, I will dive a bit deeper. Analyzing the effects of this evolution on people and companies I will explain why it is an unhealthy and unsustainable evolution.

Effects of exploding complexity on people

Before understanding the effects of exploding complexity on companies, it is important to understand its on the affected people first. From what I see, most people – no matter if in development or in operations – feel overstrained with the current abundance of concepts, tools and technologies. It is already impossible to know all these topics and its variants just superficially, let alone in depth.

Typically, this creates a constant feeling of stress, a feeling that you constantly need to pick up more new stuff than you can master, even if you would learn 24/7 and do nothing else anymore.

Additionally, the same people already experiencing a constant feeling of stress due to the abundant stream of ever-new concepts, tools and technologies usually are additionally kept in a constant state of efficiency haste: No time to think, no time to learn something new, just get the next task from your never-ending task list done as fast as possible – with just enough people in development and operations that things will not break down immediately. This adds another layer of stress due to constant delivery pressure.

This means, people are under constant stress to get arbitrary tasks done efficiently from never-ending lists. At the same time they suffer from a severe cognitive overload due to concept, tool and technology explosion causing even more stress. As they basically have no time to become familiar with all the new stuff, let alone to learn them in depth, these people often feel helpless in a race they cannot win – which reinforces the perceived stress.

As a consequence, people experiencing this kind of stress develop “survival patterns”, ways to deal with the situation, evading the stress as good as possible. The most common patterns are:

1. Knowing (almost) nothing about everything. This usually means that you try to understand just enough of a new topic at the moment you need it to get the next pressing task from your list done as fast as possible. We often refer to this as “Stack Overflow development” or “Google-driven architecture”. ¹

   The procedure is always the same: You have a problem. You search the Internet (or the existing code base) for a possible solution. You more or less blindly copy the instructions and tweak the solution provided until “it works”. You do not try to really understand the underlying concepts and what it means for future system evolution, you just try to evade the pressure of today.

   A variant of the same pattern is the “hype-driven architect/lead developer”, jumping from one new “cool” topic to the next, just knowing enough to dazzle their environment with their alleged expertise, always leaving their projects quickly enough that they are gone before the mess they created becomes apparent ².

   There are several other variants how people try to get along with superficial knowledge of the concepts, tools and technologies they use, but the pattern should be clear.

   While the yearning to muddle through this overstraining situation with just some superficial knowledge is very understandable from the affected people’s point of view, it unintentionally worsens the problem:

   You face an explosion of complexity regarding the concepts, tools and technologies that you need to build your solutions upon. This overstrains you. As a response, you add new stuff you do not really understand to your solutions, which adds even more unwanted complexity. Now you feel even more overstrained.

   In the end, this behavior results in a vicious cycle leading to premature deterioration of the software. Software becomes harder to understand, to maintain, to change, to test, to deploy, to operate – i.e., maintainability and runtime behavior suffers.

2. Hyper-specialization. This means that you try to focus on an area small enough that you actually master it. While this solves your problem, it creates a new problem on the team level: For a project with the typical technical complexity I described before, you need at least a dozen experts, often two dozens or more of them.

   It is not only hard to provide all the required experts whenever they are needed. You also hit the limits of direct communication and collaboration, which means that collaboration will suffer. Either you leave collaboration unstructured. This means that arbitrary collaboration clusters will form and vital information will not reach the place where it is needed.

   Or you structure and control communication and collaboration between the people involved, usually by imposing a structured development process including a rigid team division and communication flows – often only exchanging written documents without direct collaboration. This approach also leads to communication flaws and means that vital information will not reach the place where it is needed.

   Additionally, hyper-specialization also means that people tend to rigidly exclude topics outside their domain of expertise. They develop a jargon that is only understood inside their domain of expertise. People with different specializations – while speaking their own jargon – do not understand that jargon.

   If you ever had a situation where, e.g., people from IT, business departments, finance and marketing had to work together you know how hard it usually is to make people with different jargons understand each other. Very often, you actually need sort of interpreters who translate from one jargon to another.

   With hyper-specialization you additionally have the same problem multiple times inside your IT project – which also means that team collaboration will suffer severely and vital information will not reach the place where it is needed. There are several more downsides to hyper-specialization in IT worth exploring ³.

   But for the moment it is sufficient to understand that it compromises communication and collaboration. This leads to non-coordinated changes of the solution which in the end again results in bad maintainability and runtime behavior.

3. Blast from the past. This means that you try hard to avoid new stuff and cling to what you already know. The accompanying reasoning is that something that worked well in the past will also work well today. While this reasoning is not bad per se, it often deprives you from using (novel) options that might offer better ways to solve the problem at hand.

   Additionally, it neglects the fact that the world outside of IT changes over time, that needs, demands and expectations towards IT change over time. Thus approaches that worked well in the past might not work well anymore as they fail to address changed needs, demands and expectations.

   E.g., 15 years ago a server-side rendered web page, requiring a round-trip for every update was perfectly fine in most places. Unfortunately, this approach does not take mobile devices into account which are responsible for the majority of Internet traffic today. Additionally, user expectations changed a lot after the rise of Single Page Applications (SPA).

   As I wrote before, this pattern is not bad per se. Quite often the newest bleeding-edge hype is not the best way to solve your problem and a proven approach will be the better choice. Yet, this approach will also lead to suboptimal results not satisfying the demands of users and customers if taken too far – and the pressure to pick up newer concepts, tools and technologies will continually rise, i.e., the stress is still there, just in a different form.

4. Becoming “opinionated”. This often happens if hyper-specialization meets some specific character traits: The less harmful variant is a person who specialized in a specific topic and starts to push their topic of expertise as the solution for everything. E.g.: Everything needs to be a microservice. Everything needs to be a SPA. Everything needs to use, e.g., Cassandra or Kafka. Everything needs to be event-based. And so on.

   To suppress the (often valid) discussions regarding the appropriateness of the approach regarding the problem, the person then calls the approach “opinionated”.

   The more harmful variant of this pattern occurs if a person with a craving for recognition, but only superficial knowledge more or less accidentally creates a solution that works (or copies a solution from another place where it worked). The person obviously cannot judge if the approach fits the given problem or not due to their superficial knowledge.

   They also do not have the knowledge to come up with a different solution, but want their solution to be implemented by all means. Therefore they uncompromisingly insist that the solution is appropriate, again calling it “opinionated” to suppress the discussions.

   It should be clear that this pattern usually leads to suboptimal results due to an often severe mismatch between problem needs and solution capabilities. More often than not bad maintainability and runtime behavior will be the consequence.

5. Giving up. This means that you feel so overwhelmed from all the new stuff that floods you every day that you eventually give up and do not even try to come up with a strategy to cope with it. You somehow try to fiddle through hoping not to get detected, often without any hope that things might become any better.

   Of course this is not a sustainable pattern for the person affected by any means, and the solutions they create typically are far from being good. This again leads to premature deterioration of the software. Software becomes harder to understand, to maintain, to change, to test, to deploy, to operate, i.e., maintainability and runtime behavior suffers.

6. Burning out. This is the last pattern I want to mention here. You try extremely hard to live up to the felt expectation that you need to understand all those topics in depth – which of course is impossible. Eventually, the continuous extreme stress will break you. You might create great solutions for a while before you break. But yet it should be clear that this the least desirable pattern of all.

   Also from a totally dispassionate company point of view this pattern is not desirable. You lose very capable persons, often top performers, usually for a long time or even completely. Relevant project knowledge is lost. The solution quality suffers due to the loss of knowledge and experience, leading to reduced maintainability and runtime behavior.

A vicious self-reinforcing cycle

As I wrote before, these are only the most common patterns. There are of course more patterns that you can observe. All of them have in common that they eventually lead to a decline of software quality which makes things even worse. To illustrate the effect:

• Systems become brittle. Bug and error rates go up. Availability in production goes down. Stress increases. In turn complexity increases due to hasty fixes made under pressure.
• Maintaining, changing and extending systems becomes a nightmare due to excessive complexity. The effects of changes cannot be anticipated. Bug and error rates go up. Availability in production goes down. Stress increases. In turn complexity increases due to hasty fixes made under pressure.
• Operations becomes a nightmare as non-functional requirements are not met. Quick fixes under pressure are needed as revenue depends on it. Bug and error rates go up. Availability in production goes down. Stress increases. In turn complexity increases due to hasty fixes made under pressure.
• Development becomes slow. Trying to apply changes in an overly complex system makes you slow. Still lots of errors happen. Customers and users are dissatisfied. Important changes pile up. Pressure goes up. Bug and error rates go up. Availability in production goes down. Stress increases. In turn complexity increases due to hasty fixes made under pressure. ⁴

And so on. Ultimately, we end up with a vicious self-reinforcing cycle:

1. IT becomes overly complex in an environment of constant delivery pressure
2. People feel constantly stressed and overstrained
3. They try to evade the stress by applying “survival patterns”
4. The patterns lead to a deterioration of quality and increase of complexity
5. Bug and error rates go up. Availability goes down. Pressure increases
6. Go back to 2

To escape that cycle people often try to clutch at every straw they see, hoping for relief but usually making things just worse:

• They try harder to “Do the wrong thing right”, i.e., do their work more efficiently without addressing the underlying problems.
• They push “Agile” harder without addressing the underlying problems (also see footnote #2).
• They jump on the next hyped technology bandwagon, creating even more bloated solutions, expecting that they will offer a way out of the cycle, adding to the underlying problems. ⁵
• They introduce quality theater, e.g., radically enforce clean code, unit testing and alike, resulting in more work without addressing the underlying problems.
• As the business departments do not see their problems solved, they come up with ever more requirements hoping that they will eventually solve their problem, fueling complexity without addressing the underlying problems.
• Managers go for more controlling, resulting in more work without addressing the underlying problems.

And so on. I could go on for a long time, but in the end we have a self-reinforcing cycle, started by complexity growing over a tipping point, often accelerated by a pointless efficiency haste. ⁶

Summing up

In this post we have seen that excessive complexity is very unhealthy for people affected by it as well as for the companies they are working for. Excessive complexity, especially if combined with constant delivery pressure acting as an accelerant, leads to people feeling constantly overstrained, continuously experiencing stress.

The typical “survival patterns” to escape that feeling of stress tend to make things even worse over time, creating a vicious self-reinforcing cycle. Additional “sheet anchor” activities that people often clutch at, hoping to relieve the pressure usually do not address the underlying root causes, also making things worse over time.

This is very unhealthy for the people affected by it and not rarely it ends in burnout or other serious health issues. It is also very undesirable from a company’s point of view. Not only their employees get sick, they also end up with IT systems that are buggy, hard to maintain, extend, test and deploy, and unreliable in operations – quite the opposite of what they want to achieve.

To sum things up:

Excessive complexity is unhealthy for the people and undesirable for the companies affected by it. Therefore, we should work hard to avoid it.

In the next post, I will start to discuss essential and accidental complexity as we can only get rid of accidental complexity, i.e., complexity not needed to solve the problem at hand. But more of that in the next post. Stay tuned …