Simplify! – Part 5

In the previous post, we looked at the different types of accidental complexity that help us to identify the places where to look for unnecessary complexity.

I also wrote that before starting this search, it first makes sense to understand better the origins of the current excessive complexity. How did it emerge? Understanding this evolution provides us with the final piece we need to understand before we can discuss countermeasures to simplify IT again.

From what I see there are many factors contributing to this evolution and many of them influence and reinforce each other. This makes it hard to describe the factors completely independently. Still, I try to sort them apart as good as possible.

To provide a bit of a structure, I move “from the outside in”, i.e., I begin at the market level and then move from company over project, architecture, tools and technologies to coding, plus some general IT-related observations at the end.

Initially, I planned to first describe the evolution with all its drivers completely from the aforementioned perspectives before moving on to the question what we can do to reduce the complexity. Unfortunately, while writing I realized that this would have resulted

• either in a very long post that nobody would read,
• or in several shorter posts which would have meant a long wait time before seeing the first remediation option.

As both options appeared a bit underwhelming to me, I decided to sort the posts a bit differently. Starting with this post, each post will discuss a single perspective in full: the evolution that lead to the current situation and the remediation options.

As the different perspectives usually cannot be discussed in full isolation, I will also add the cross-references to related perspectives where needed.

Changing markets and digital transformation

Let us begin at the very outside: the markets and customers that exert pressure on the companies. Many of the new concepts, tools and technologies that are haunting us now were sort of responses to changing markets and the digital transformation. As I described in this post, most efficient markets meanwhile have become post-industrial. Post-industrial markets create a lot of dynamic and uncertainty which means that companies living in such a market need to

• Go fast
• Gather feedback fast
• Learn and adapt fast
• Offer highly available products and services ¹

In short: They need to adapt fast to ever-changing needs without compromising quality (as perceived by their customers).

Digital transformation on the other hand means that IT becomes a more and more indispensable part of the business. IT no longer is just a “cost center”, but has become an enabler and vital ingredient of all services and products companies offer. The interactions with the customers as well as the products and services themselves are supported by or even completely driven by IT.

Therefore, the IT departments also must implement a post-industrial working mode that enables them to move fast without compromising quality.

Doing more of the old

This poses a big challenge on the affected IT departments. They need to facilitate everything the company’s business side needs from them to stay competitive in a highly dynamic market. This means that at least they need to:

• deliver in shorter cycles (ideally multiple times a day)
• dramatically shorten their lead times (that start with an idea and end when customers can provide feedback on the IT-driven result)
• enable flexible business experiments at any time (including testing multiple variants of an idea at the same time)
• deliver feedback from the customers (by watching usage and adoption patterns on a business level and providing easy-to-use feedback channels to the customers)
• pick up new IT trends to enable novel digitally fueled business ideas (and co-create them with the business experts)
• deliver high quality all the time (as experienced by the customers – including high availability, responsiveness, etc.)

If you are the manager of a traditional IT department, this gives you a big headache. How to get there? It seems to be impossible with the processes and practices you have in place:

• Complex budgeting and project approval processes with a long lead time
• Elaborate, detailed and rigid software development processes with lots of activities, long paths from start to end, many handover points including lots of artifacts, several layers of hierarchy and extensive reporting
• Strict separation between Plan, Build and Run with narrow handover points and long queues in front of them
• An (enterprise) architecture department that defines strict and rigid guidelines regarding allowed technologies and development practices, and that needs to approve all changes to these guidelines
• Complex rule sets in operations due to ITIL or alike that require complex procedures to take new stuff live, and that make changes requiring new concepts, tools or technologies cumbersome

… and so on. In short: There does not seem to be any reasonable path to get from the status quo to an IT that satisfies the aforementioned requirements. It seems to be impossible!

To be fair, the effects of the changing markets and the digital transformation did not become apparent at once, but slowly over time. This means that the affected IT managers did not see the issues at plain sight as I just wrote it down.

Instead, they saw a creeping change for the worse – a bit higher delivery pressure here, some new technology needed there, and so on: going from 2 releases per year to 4 releases, adding support for mobile devices, first just a standalone application, then with a bit of integration in the existing systems, and so on. Step by step.

Probably, most mangers would have responded quite differently if they would have seen the accumulating demands in their final clarity. But observing the changing demands step by step, it is easy to see that most managers responded by doing more of the old and “proven”, the things they were used to do. E.g.:

• Doing more rigid investment planning, hoping for better cost control
• Creating more detailed plans with more controlling and micromanagement, hoping for more predictability and less risk
• Standardizing tools and technology in development further, hoping for more efficiency
• Creating more detailed guidelines how to do things including more quality gateways, hoping for less variance and more efficiency
• Reducing the set of allowed technologies in operations even more, hoping for more quality

Knowing the aforementioned target state as we know it today, this approach of doing more of the old seems absurd: How can one think that it is possible to improve speed, throughput and flexibility by several orders of magnitude simply by pressing harder and doing more of the old stuff?

Yet, this is the difference between sitting in a tub with water that is gradually heated up and getting into the tub that is already filled with too hot water. While you would refuse to get into the tub in the second case, you would very likely still sit in there in the first case even if the water would have reached the same temperature. Well, sure it feels uncomfortable but it is not that much warmer than it was a minute before, is it?

Still, what most managers lacked to do is to step back from their daily treadmill to realize that current state and target state of their IT are so far away from each other, that demands meanwhile went up by several orders of magnitude that stepwise improving efficiency by doing more of the proven definitely will not solve the problem.

In other words:

If you are stuck in a traditional IT while facing the demands of a post-industrial IT, just doing more of the old is not an option.

You need to rethink your IT from scratch.

Problem multipliers

To make things worse, some other issues acted as accelerants. On a general company level, the fact that most decision makers neither understand IT in itself nor the role of today’s IT leads to a lot of bad decisions that ultimately made the situation worse instead of better. I will discuss this topic and its effects in detail in the next post of this series.

Additionally, lots of decision makers hoped that “Agile” will solve all their problems and thus jumped that bandwagon. This is understandable as many of the “Agile” advocates promised unprecedented productivity boosts. Unfortunately, it turned out differently than expected: The agile ideas met a strictly industrial mindset resulting in the same old industrial practices, focused on (small) efficiency gains, just disguised behind some new terminology.

Also some other concepts were blindly copied, like, e.g., DevOps, also with a deeply industrial mindset which lead to similar effects like the “Agile” transformations. I will discuss in one of the next posts of this series how these “Agile” and “DevOps” cargo-cults actually increased complexity instead of reducing it.

Increased accidental complexity

So, we are still stuck with an industrial mindset facing the continuously rising pressure of post-industrial markets and digital transformation. Efficiency-focused thinking trying to address uncertain effectiveness. This is a big driver of accidental complexity, because the typical responses of the decision makers tend to make things worse instead of mitigating them. E.g.:

• Complex budgeting processes lead to lots of unnecessary requirements.
• Inhibiting investments approval processes lead to lots of badly maintained and outdated parts of the IT landscape, blocking lots of people and money, impeding required system landscape upgrades even more, increasing accidental complexity.
• Maximizing utilization of IT employees leads to reduced productivity ², impeding required cleanups of the IT landscape.
• Short-sighted cost savings lead to overly complex software with accidental complexity multiplying up.
• Blindly copying concepts lead to overly complex tool and technology landscapes regarding the actual problems.
• Rigid platform limitations lead to overly complex software solutions trying to compensate the shortcomings of the platforms.

And so on. The list could go on for a long time (and I will discuss some of those topics in more detail in the next posts). In short, we can say:

IT departments are under the pressure of responding to new drivers and forces resulting from post-industrial markets and digital transformation.

At the same time the organization and decision makers around them, not understanding post-industrial markets and digital transformation, unintentionally massively impede any adequate response. On the contrary they make things worse, driving accidental complexity.

Help people to understand

After analyzing how not understanding the consequences of post-industrial markets and digital transformation leads to a lot of accidental complexity (not to mention the frustration of the people involved), the obvious question is:

What can we do to improve the situation?

The simple response would be: Tell them to learn about post-industrial markets and digital transformation beyond the buzzwords – “them” being the decision makers. Yet, while this answer is tempting, it is merely passing the buck. And worse: it creates an “us an them” thinking: “I did everything fine. They have to move.”

Personally, I am not a fan of those “us and them” settings. Sure, it is a general problem if the decision makers do not really understand the consequences of external drivers and thus a bit unimaginatively just respond the same way they always did. Still, just passing the buck is not a constructive solution.

Thus, what can we do to improve the situation?

Well, for starters, we can help the affected people – which are, by the way, a lot more people than just the decision makers – to understand changing markets, the resulting demise of certainty, the consequences of digital transformation and the new role of IT better.

Yes, this means a lot of talking, repeating your message, lots of tedious discussions, lots of struggle for little victories and more. The good part of the story: from what I observe, most people are a lot more willing to listen these days than they were in the past.

I started discussing post-industrial IT in 2013, that we need to rethink IT. Back then people smiled about it and gave me that “Dream on”-look. In 2015, the smile was gone. In 2017, it was replaced by a puzzled, nervous look. Today, the same people who smiled in 2013 are way beyond nervous, knowing quite well that they missed to start moving when they needed to.

Admittedly, knowing that you need to do something, does not mean that you will do it. There are still habit and fear of the unknown as the big inhibitors of change.

On the rational side people first need to understand the “why” and the “what” before they are willing to move to the “how”, i.e., changing anything. That is what we need to support. We must help the people to understand the forces that drive change. We must help them to understand the options they have and what those options mean in terms of costs, effects, time and risk.

Personally, I also think that we should stop hiding the complexity of IT. I cannot remember the number of times when somebody told me boil complex topics down to a few simple bullet points when talking to a higher decision maker. Usually, I did. But meanwhile I think it was not only wrong, but also unfair to the decision maker. If you hide all the complexities from the decision makers and give them the feeling that they have to decide about something trivial, how should they make the right decision?

Therefore, while always trying to have my respective target group in mind and to explain things in a way comprehensible for them, I stopped hiding the complexity of the task that needs to be decided.

Go for continuous improvement

On the more emotional side, I recommend creating an environment where change does not feel that scary. Even if people always pretend that they are “professionals” that act and decide “purely rational”, in reality our decisions are a lot more gut-driven than we like to admit. Expecting a big change without a way back from people typically makes them feel very uncomfortable as they cannot grasp their personal risk of the requested change. Humans tend to assess an unknown risk as a very high risk – and thus respond with rejection.

Therefore, it is important to create a setting where

1. a single change step is not too big to keep the perceived risk at bay
2. people can test the change before they need to commit to it to to actively strip the uncertainty multiplier from the perceived risk.

There are multiple ways of providing such a setting. My personal recommendation is to introduce a continuous improvement process as it provides the properties I just described. The typical pattern of such a continuous improvement process is:

1. Propose a change, that is small enough that the people affected can still digest it
2. You agree to try it for a limited period of time to see if it has the effects that you think it has
3. Everyone works in the new mode for the agreed upon period of time
4. At the end of the period you meet and decide if you want to keep the change or revert to prior state
5. Go back to 1

You can also limit the experiment (step 3) to a limited number of people or activities, i.e., do a “pilot”. This would resemble the PDCA cycle. The key point is that the change is not final when people start implementing it, but has a “test period”.

This reduces a lot of resistance from risk-averse people. If you give them the opportunity to test a change and get familiar with it before having to decide if they want to commit to it, they do not need to feel anxious about the change anymore and thus tend to resist a lot less.

A continuous improvement process means that you do not test and implement just one or a few little change steps, but that you implement change continuously, i.e., that over time you integrate it in your work culture.

Of cause, this also means that you need some good metrics that help you to understand if your activities actually improved anything with respect to your goals ³. But the first thing is getting a change process running.

Summing up

This was a long post discussing the general effects of not understanding post-industrial markets and digital transformation ⁴. From what I have seen this is a root cause of many wrong decisions that brought IT in the bad situation where it is right now, being overly complex with tons of accidental complexity.

This is also the reason why I did not use the little accidental complexity framework, I developed in the previous post of this series. We do not see the effects of not understanding post-industrial markets and digital transformation directly, but they result in a lot of decisions that in turn drive accidental complexity.

Maybe you will not find the countermeasures I offered satisfactory. As the issues I described here affect accidental complexity only indirectly, I have to admit that I do not see any simpler or more effective countermeasures than the ones I described ⁵. Yet, as effecting a change on this level has such a huge lever, we need to do what we can to trigger any improvement.

In the next part I will continue at the company level, discussing the effects of not understanding IT in itself, especially the nature of software, and not understanding the role, IT has today. So, stay tuned … ;)