Leaving the rat race - Part 3

In the previous post we discussed that we are by far not as “innovative” as we claim to be, that most of the innovation puffery comes from a hype industry, trying to lure and force people into buying their products and services.

In this post, we will discuss that most of the “innovation” hype is but distraction from our actual problems.

The distraction from the actual problems

If we take a critical look at all the “innovations” we are confronted with on a monthly, weekly, daily basis, we see that most of them promise to solve some of the enduring problems we have in IT for a long time:

• Reducing complexity and thus cognitive load
• Improving changeability and evolvability
• Increasing productivity and/or reducing time-to-market
• …

Often these promises are made indirectly, e.g., by promising improved reusability which would massively improve productivity.

But no matter if the promise comes in a disguise or not, the promise most of these new tools, technologies and approaches make is to magically solve some of the enduring problems of IT: They promise to cut through overwhelming solution complexity, to make changes a breeze or to unleash developer superpowers that make them ten times more productive.

And none of the promises was kept ever.

Which is not a surprise because all these new tools, technologies and approaches do not address the actual problems we face in IT.

Admittedly, they might feel fresh for a moment and things may feel a bit better temporarily. But as soon as the freshness effect wears out, we see that in the end nothing really changed – except for another tool in our tool cemetery, another brick in our ever-growing complexity tower.

But why is it this way?

Because all these tools, technologies and approaches are but distractions from our actual problems, those problems that actually lead to increased complexity, crippled changeability, reduced productivity and long time-to-market duration.

The actual problems

The actual problems are known for a long time. Many of them are known for more than 50 years meanwhile. Unfortunately, an easy solution for these problems does not exist. Solving them is hard work and often means deviating from the seemingly simple and obvious path.

As hard work and acting against established habits does not sound very inviting, most people shy away from it. Instead, they try to find an easier solution – ideally a “silver bullet”, automagically solving all the hard problems. Not only, but especially in IT, we try to find these “silver bullets” in tools, technologies and new approaches.

If we face a problem, the default response is looking for a tool that promises to solve it. If a tool does not (yet) exist, maybe a technology will solve the problem, or at least some standardized approach with simple rules we just need to follow. As most people in the world, we also look for simple answers to complex problems and history shows there is basically nothing that can dissuade us from this habit.

The product vendors and consulting companies understand this habit, too, which lead to the hype industry we discussed in the first post of this series.

But what are the actual problems?

Here I will briefly sketch the most important ones I am aware of (some of them I already discussed in more detail in former blog posts; some of them I will probably discuss in future posts):

• Functional design – We still have not learned how to do good functional design. Some good ideas were developed in the late 1960s and the early 1970s. But since that time basically no progress has been made. Also the currently widely popular Domain-Driven Design (DDD) basically is just a collection of ideas that are way older than the book by Eric Evans.

  But even if we should have understood in theory how to create a good functional design, it is still hard work to apply these concepts in a concrete context. I have seen quite some people who knew the DDD patterns by heart, but still created traditional SA/SD-type designs when trying to apply them.

  Actually, most designs I see are still a mix of SA/SD and E/R modeling. Of course, we renamed it several times to give it a fresh look. For a while, we called it OO design. Currently, we call it DDD. But if you look at it, very often it still is SA/SD, just using newer terminology.

  While this kind of design has value in certain places, it is not well-suited for many of our contemporary challenges because it often fosters tight coupling at the wrong places, leading to solutions that are more complex than needed and often hard to change.

  We have not solved our functional design problems for more than 50 years meanwhile. But this is the key to solution complexity, to cognitive load, and derived from it to solution evolvability and developer productivity. Solving this problem alone would probably solve 99% of the problems, we hope to solve with all those “silver bullets”.

• Architectural work – Architectural work sucks in many places. Sorry to be so harsh, but a lot of architectures I have seen were barely connected to the problems they should help to solve. Surprisingly often, I have seen complete architecture designs before the first requirement was captured. And not surprisingly, most of these architectures looked a lot like a collection of the hypes fashionable at that time.

  But that only leads to a lot of accidental complexity (see this post for a distinction between essential and accidental complexity) which increases cognitive load, impeding solution evolvability and developer productivity.

  As an aggravating factor, we tend to admire people who create unnecessary complexity. If you design an overly complex architecture, people tend to look at you in awe: “How smart (s)he is! Wow! I would not have been able to come up with such a sophisticated architecture.”

  If you come up with a very simple architecture instead, people often turn up their noses at you: “That’s trivial! Everyone can do that. We don’t need an architect for that!”

  The problem is that the opposite is true. Every simpleton can can come up with an overly complex solution design. That is easy. But it requires a lot of hard work (and persuasive efforts) to come up with the simplest solution possible that suits the needs of the problems. Shedding off accidental complexity from a solution design is hard mental work.

  The fact most architectural work is not worth its name adds to the aforementioned problem that we have not solved our functional design shortcomings. As a consequence, most of the times we end up with overly complex architectures lacking good functional separation of concerns, resulting in increased cognitive load and reduced changeability, productivity and time-to-market.

• Feedback loops – Most organizations suppress feedback loops that are urgently needed to align problem and solution domain. A solution domain perfectly aligned with the given problem domain would be an IT landscape just consisting of essential complexity without any accidental complexity. But without proper feedback loops you will not even get close.

  First, you need a feedback loop between requirements and solution design. Often, people write requirements that are at odds with the existing design. This is not malicious intent, the requesters simply do not know better. With a proper feedback loop, the solution designer could discuss the requirements with the requester and point out their unintended consequences.

  Very often they would be able to modify the requirements in a way it provides the same outcome without being at odds with the existing solution design. And sometimes the solution designer would learn that she missed some relevant business needs when originally designing the solution and thus decide to adapt the solution design – but based on much better input than she would have without the feedback loop.

  Either way, such a loop would avoid a lot of unneeded complexity. But very often it is not established.

  We also need feedback loops between solution design, development, information security, compliance and operations to make sure nothing essential is overseen upstream in the process, creating additional complexity or even defects downstream. ¹

  Finally, a feedback loop between the users of the solution and the requesters who created the original requirements is needed to help the requesters understand if their requirements meet the needs and expectations of the users. Otherwise, the solution will contain more and more features the users do not want, i.e., more and more complexity without value.

  Without all these feedback loops, we are doomed to create a lot more complex solutions than we need to, leading to increased cognitive load, impeding solution evolvability, developer productivity and time-to-market.

• Short-term efficiency thinking – Most organizations and processes put their (sole) focus on maximizing short-term efficiency. The leading question is how much output can be achieved now. The decision between implementation alternatives is based on immediate costs and efforts. Any long-term consequences are neglected and hero is who can shed a few hours now – even if the shedding will most certainly result in massively increased efforts and production bugs in the future.

  This focus on short-term efficiency leads to quickly deteriorating designs and implementations. Maybe the original design was good with respect to the given needs. But all the “workarounds” added over time to “improve” efficiency, to shed a bit of time here and there diluted the design more and more, bypassing its original ideas and principles. Eventually the system feels like a big ball of mud – ironically with everybody trying to do their best, just guided by the wrong shortsighted forces.

  This also drives more complex solutions, leading to increased cognitive load, impeding solution evolvability, developer productivity and time-to-market.

• Ignoring the needs of software – Software needs to be changed continually to preserve its value. This makes it very different from most other goods. Still, people tend to compare software to physical goods like cars or houses, just to name the two most widespread (flawed) comparisons.

  And as written in the further course of the aforementioned blog post, continuous change means continuously growing complexity unless explicitly tackled. This means the complexity of software must be explicitly managed over its complete lifecycle or it will proliferate in an uncontrolled way.

  Unfortunately, this essential need of software is ignored most of the times. The fact that software is invisible, thus basically unvisualizable and lacking a good “coordinate system” (at least in the relevant dimensions), makes it even harder. ²

  This is one of the reasons why so few organizations understand that sole short-term efficiency thinking is venomous for software. The consequences of this behavior remain invisible when we would need to counteract it and thus the required counterforces cannot establish themselves – resulting in more complex solutions, leading to increased cognitive load, impeding solution evolvability, developer productivity and time-to-market.

Summing up

These were some of the enduring, unsolved problems we still face, the actual problems leading to increased complexity, crippled changeability, reduced productivity and long time-to-market duration.

All the tools, technologies and approaches, promising reduced cognitive load, better software evolvability, higher developer productivity and so on, are in the end merely distractions, the ever-repeating hope we do not need to tackle our actual problems in IT.

The issue with our actual problems is that they are hard to solve. A silver bullet does not exist. Solving these problems means hard work, letting go of seemingly good practices, rethinking our habits and value systems from scratch. But that is hard, very hard. And you will face a lot of resistance because this means change.

Jumping for the next tool, technology or approach is a lot easier. And it does not result in fierce resistance but is a widely accepted practice. To adapt a well-known phrase: “You do not get fired for suggesting a new tool.”

But you could get fired for suggesting a whole new way of approaching problems because then it is your fault if things do not work out as expected – even if it is not your fault. But it was your idea. Taking all that into account, it is very understandable most people take the easier and less risky path.

But over time this behavior created a new category of problems like FOMO ³, the hype industry, people stressed out due to seemingly continuous “innovation” and more – without solving any of the existing problems.

At least I think this is a bad trade.

In the next post, we will look at the actual strengths of humans and why the hype industry tends to play against it. Stay tuned … ;)

PS: I will pause publishing posts during the holidays. The next post will be released in January. I wish you all a wonderful time and a great start into 2022!