Simplify! – Part 15

Hooray! This is the last post of the “Simplify!” series. As I wrote before: I would never have expected that it would become that long. Initially, I expected 4, maybe 5 parts. Most likely I would not have dared to begin with it if I would have known. Anyway, we have arrived here – at last!

Before I add a few final considerations, let us briefly recap what we have discussed so far:

• Part 1 – Software has become indispensable in our everyday business and private life. If the software does not work as intended, we have a problem. At the same time, complexity in IT explodes and we are drowning in ever-growing complexity in IT – a worrying combination.
• Part 2 – The complexity explosion stresses affected people out. As a result they develop counter-productive “survival patterns”: Going for superficial knowledge. Hyper-specialization. Ignoring developments. Becoming “opinionated”. Giving up. Or burning out while trying to keep up. Those patterns lead to vicious self-reinforcing cycles that lead to even more complexity.
• Part 3 – Complexity needs to be divided in essential and accidental complexity. Essential Complexity is inherent in, and the essence of, the problem (as seen by the users). Accidental Complexity is all the rest —- complexity with which the people affected would not have to deal in the ideal world. While we cannot eliminate essential complexity, we need to work hard to avoid accidental complexity.
• Part 4 – We need to distinguish different kinds of accidental complexity: Due to unnecessary requirements not adding value to the solution. Due to overly complex platforms regarding the given problem. Due to a lack of adequate platform support regarding the given problem. Due to overly complex software regarding the given problem.
• Part 5 – Accidental complexity on a market level. Old response patterns to changing market realities lead to a lot of accidental complexity. Many companies face post-industrial markets and try to address changed markets demands by intensifying old industrial response patterns.
• Part 6 – Accidental complexity on a company level. Not understanding IT (especially its primary “material” software) and the role of IT today lead to a lot of accidental complexity, either by imposing counter-productive practices on IT or by blindly copying concepts from others without understanding them.
• Part 7 & Part 8 – Accidental complexity on a project level. Degenerated budgeting and IT staffing processes. Inward-bound focus of the organization resulting in a loss of market focus. Neglecting uncertainty reinforcing the situation. “Agile” and “DevOps” cargo cults, reinforcing industrial practices in a new disguise. The “high utilization means more productivity” fallacy. Rigid project constraints for the wrong reasons. All these and more lead to increasing accidental complexity.
• Part 9 – Technology explosion as the result of an ongoing, not yet completed technology evolution cycle. FOMO and the tendency to blindly copy concepts as reinforcing drivers. And so on. Massively increased accidental complexity as a result.
• Part 10 & Part 11 – Accidental complexity on an architectural level. Unresolved design deficiencies. Black holes and strange attractors crippling evolution. Architecture narcissism leading to bloated designs. Neglecting the business domain and focusing on technology only, and vice versa. The habit of DIY, combined with misconceptions regarding OSS. Missing the technology evolution. All these and more lead to increasing accidental complexity.
• Part 12 – Accidental complexity on an implementation level. Breaking existing designs. Going for implementation shortcuts. Design by Stack Overflow. Show-off implementation. Keystroke theater. Quality theater. All these and more lead to increased accidental complexity.
• Part 13 & Part 14 – An IT landscape consisting of layers of systems never cleaned up. Missing responsiveness of IT leading to solutions bypassing the IT department. Fear of decommissioning old systems. Missing the technology evolution and when to replace complex custom-built solutions. The “no-time to clean up” fallacy. The big clean-up initiative, trying to fix the existing mess “once and for all” with a single approach. All these and more lead to increasing accidental complexity.

We have augmented this series with 3 posts about OSS, the first one discussing the rise and benefits of OSS, the second one discussing widespread misconceptions regarding OSS and the third one discussing the changed role of OSS in a cloud-native world.

Additionally, we have discussed the continuous amnesia issue of IT, i.e., that IT as an industry tends to continuously forget everything it has learned. As a result, it tends to make the same mistakes over and over again, also making to harder to improve with respect to the complexity challenges we have.

Including this post, these were 19 posts! As I wrote before: If I would have known that in the beginning … ;)

But even if this series has become a lot longer than I ever expected, I think it was worth it. The simple fact that it took me 15 posts just to describe the most important aspects of the topic shows, how messed up the situation in IT is, how relevant the topic has become.

In a tad melodramatic way we could phrase it:

Simplify or die!

And even if this sounds a bit exaggerated, I think it is closer to reality than it appears at first sight:

• IT Complexity creates competitive disadvantages, compromises the ability to innovate, and worse.
• IT Complexity stresses people out, leading to churn and burnout.
• IT Complexity threatens the functioning of companies.
• IT Complexity can even threaten the functioning of our society in the worst case – remember: Also our private lifes depend more and more on working IT.

To make things worse, the usual response patterns to these challenges are counter-productive, often leading to more complexity – a vicious reinforcing cycle that we need to break.

I discussed several ideas how to address the drivers of accidental complexity on the different levels. Not surprisingly, many of them require a massive rethinking of IT, what IT is, what its role is, how to run it and how to interact with it. Otherwise, we would never have arrived in such a mess in the first place.

As always, it is hard to establish new ways of thinking and acting. People dislike to give up familiar habits, and cultural changes are hard. Nobody will say: “Great that you came along! We now see that all our practices were wrong! Let us quickly change all of them!”

As with all change processes, it will be a long, tedious way and take a lot of time. You need perseverance. You need good arguments. You need intuition and tact to address the different people in the right way at the right time. But this is nothing special only applying to this topic. This is the default for all change processes. So, maybe a good change agent can help.

For the arguments, the rational part of the change, I tried to provide you with some reasoning why it makes sense to do things differently. I hope it will help you on your way, if you should decide to simplify your environment.

General recommendations

I would like to finish this series with some general useful reasoning tools. They are not limited to tackling accidental complexity. They work in many places, but they are also useful when tackling accidental complexity:

• Ask “Why”. Why should you use a new concept, tool or technology? Which part of the problem does it address? Do we need anything new for that or is it already addresses? Is it really needed or do we rather redirect the river already running through our fields to be able to build an aqueduct ¹? We need to ask “why” to create the required focus and direction for our decisions and actions, e.g., if we really need a new technology or not.
• Think holistically. Try to see the whole picture. What does the decision mean for the developers coming after you? What does it mean for operations? How well does it integrate with the existing solutions? What does it mean over time? You need to leave your default perspective and look at your decision options from multiple angles. ²
• Apply Occam’s razor, sometimes also referenced as the YAGNI principle. If you are not sure, go with the simpler solution. Try to go with the smallest and simplest solution that does the trick. The probably hardest part is to accept the inherent uncertainty and the limits of foresight that always exist. Often, overly complex solutions are created because “we do not know what will be needed in two years”. Accept that you do not know it. Especially accept that the chances that the complex solution you intend to create will miss the still unknown future need as likely as the simple solution that you can create alternatively. Thus, in case of doubt: Go for the simpler solution.
• Implement feedback loops. A common driver of complexity and other undesired effects is the separation of people who make a decision and the people who are affected by it. You can observe this effect in many places, also outside IT. The best remedy is making sure that the decision makers are also affected by the consequences of their decisions. If this is not possible, you should at least strive to establish feedback loops between decision makers and people affected. Often decisions are made with best intentions but missing feedback loops inhibit learning how good the decision actually was, if corrective action is needed. Feedback loops can help to get rid of a lot of accidental complexity.
• Accept uncertainty. This has two facets. First, if you try to remove all company-internal uncertainty, it will create tons of overhead without actual value, i.e., accidental complexity. It is okay to tackle internal uncertainty to a certain degree. But as so often it is about the right degree, not about all or nothing. Second, if you try to eliminate external, market-induced uncertainty, you will end up with highly inefficient methods creating a lot accidental complexity. Accept that the market is uncertain. Embrace it and use it as a chance. Make uncertainty your ally. If you know better than your competitor how to play with market uncertainty, it gives you a competitive advantage.
• (Use common sense). I would love to add this one. Yet, to be honest I hesitate to do so. There are so many places where I think the best possible decision should be obvious based on “common sense”. But I also learned that “common sense” means something different for every person. So, while being correct and useful in some way, this advice can also quickly become counter-productive. Especially it not provide enough direction, I am afraid. Maybe you have a faint nagging voice in your head repeatedly asking if all the solution complexity is really needed – probably that is the closest thing to what I mean with “common sense” in this context.

These were some general reasoning tools that can help to mitigate accidental complexity at all levels. As always there are more, but I think these ones are a good starting point.

All of the these tools also help to implement the probably most important recommendation:

Learn to do the right thing. Do not only focus on doing things right.

Doing the right thing is effectiveness. It is a prerequisite for creating value. We can be as good as we want with your tools, technologies and practices in IT. If we are not doing the right things, we destroy value, and all our skills, expertise and energy are wasted for nothing. Therefore, it is vital to learn to do the right things. The aforementioned tools help you to discover them.

Doing things right is efficiency. Efficiency also has a great value. But it needs to be applied to the right things to unfold its value. You do not become world champion by running 100 meters faster than anybody else, but not in the direction of the race track. You will simply be disqualified. The same is true with applying efficiency to the wrong things. You will gain nothing from it. Instead, energy will be wasted and value will be destroyed. But if you know what the right things are, then efficiency can add a great lot of value.

Overall:

Become effective first, then add efficiency.

This will minimize accidental complexity and maximize value.

Doing it the other way round will increase accidental complexity and destroy value.

Summing up

As I wrote in the beginning of this post: This was a looong blog series about excessive complexity in IT and what we can – and should – do about it. I hope the series gave you some useful ideas to ponder, maybe even some actual insights.

I mentioned a lot of topics in the series that I want to discuss in future posts. For now I will leave it here, sit back, take a deep breath and look forward to other exciting topics – that will not require 15 posts to discuss … ;)