Forget efficiency

Should we really forget efficiency? No. But we should postpone focusing on efficiency until we are effective.

Basically, the previous sentence is the summary of this blog post and maybe it sounds obvious at first reading. But based on my experiences it is not. The biggest problem we have (not only) in IT these days is a lack of effectiveness. And what do we do to solve this problem? We attempt to improve our efficiency – amplifying our problems instead of solving any of them.

I mentioned this issue several times in previous posts, usually in the context of other topics. Taking a few steps back, I realize this is probably the root cause for many, if not most problems we face in IT today (and in many other places). Therefore, I decided to dedicate a whole post to this issue.

Efficiency vs effectiveness

Let us start with the difference between efficiency and effectiveness.

According to Wikipedia, efficiency is defined as:

[Efficiency] is the ability to do things well, successfully, and without waste.

Also according to Wikipedia, effectiveness is defined as:

Effectiveness […] is the capability of producing a desired result or the ability to produce desired output.

Simply put, we can define the two terms the following way:

Effectiveness is about doing the right thing. Efficiency is about doing the thing right.

With this definition, the first paragraph of this blog post should become obvious by all means: Unless you do the right thing, doing it better will not increase any value.

Doing the wrong thing does not create value. Doing the wrong thing more efficiently still does not create any value. It just became cheaper to not produce any value. Well, it does not become cheaper in practice. As your goal is to produce value, you will use up all spare capacity you get from your efficiency improvements to do more production, still without creating value. It is a death spiral because you invest more and more energy in not fixing the actual problem.

In short: Improving efficiency while not being effective is pure waste – and extremely painful for all people involved.

Drivers of efficiency thinking

All this should be obvious, but obviously it is not. What is going wrong?

As so often, I do not have the perfect answer. Still, based on my experience I see two core drivers for this misguided focus on efficiency:

1. The often still predominant economic ideas regarding markets stem from a time with very different conditions. Most market-oriented literature is built on ideas from the first half of the previous century (some are even older like the idea of the “invisible hand” that regulates the market). Back then, at least in the western hemisphere we faced industrial markets. These were defined by demand being significantly higher than supply. In such a setting, effectiveness is relatively easy to achieve. Unless you are doing it terribly wrong, customers will buy your products or services – because demand is higher than supply, i.e., the market is not saturated. Hence, your key to success is producing more at a lower price, i.e., to improve your efficiency.
2. The often still predominant economic ideas regarding IT stem from a time with very different conditions. Most concepts regarding software development are based on the situation, we faced from the 1960s to the early 1990s. This was the time of the “software crisis”, a time when demand for software development was significantly higher than the supply of developed software. Additionally, software development was a complicated task, not a complex one ¹. In such a setting, your primary goal is to increase the amount of software developed, ideally while lowering its price. Effectiveness was not a big issue. Do what your customers demand from you and everything is fine.

While the times mentioned are long gone, the ideas they shaped reverberate until today. For many years, efficiency was the primary goal and effectiveness was not considered being an issue. If approaches, priorities and values are repeated long and often enough, they become part of a company’s culture, i.e., they become ingrained in the company’s collective memory and shape the thinking, acting and interaction of the people involved.

It is important to understand this is not about the individual. It is about the interaction and response patterns ingrained into the company’s collective brain. Replacing people does not change a lot ². Either the new persons adapt to the commonly accepted interaction and response patterns or they are repelled by the company culture and will eventually leave.

As a consequence, we still face a predominant focus on efficiency in most companies while effectiveness is more or less considered granted. Note that this does not necessarily become clear when listening to what people say (like, e.g., “we need to focus on our customers’ demands”) but when looking at how they act (following through the commonly accepted procedures that sacrifice the customers’ demands on the altar of efficiency).

Autopoiesis as reinforcing factor

Another reinforcing factor is what Niklas Luhman called “autopoiesis”. He used this term in his system theory to describe the self-creating and self-preserving properties of systems (like a company or its organizational units).

To repeat the very short and simplified explanation of the term, I already used in my blog posts about understanding uncertainty and accidental complexity at the project level as part of my Simplify! blog series:

Systems (like companies or organizational units inside a company) form to fulfill a purpose. The purpose is defined externally – some need, demand or necessity. Over time, the systems decouple from their original purpose and become an end in itself. Their further evolution is then driven by internal needs instead of external drivers (Luhman calls such systems “operationally closed”).

This was the extremely short and simplified explanation of autopoiesis. I may discuss this topic in more detail in a future blog post. For this post the key message is that most companies and their organizational parts tend to decouple themselves from their external customers and their needs, and eventually revolve around themselves.

You may have experienced this effect yourself several times. Whenever you thought by yourself “This is not customer-oriented” (no matter if it is about internal or external customers), most likely autopoiesis struck again. Personally, I see these effects over and over again. Once you learned the concept of autopoiesis, it is impossible to unsee its manifold implementations.

When looking at efficiency and effectiveness and how autopoiesis affects them, we observe the following:

Efficiency is inward-focused. It is about optimizing internal processes. Autopoiesis reinforces this point of view.

Effectiveness is outward-focused. Effectiveness is defined by the needs and demands of the users, the customers, the market. Autopoiesis impedes this point of view.

As autopoiesis tends to be strongly developed in most companies that exist for a while, it reinforces the company’s cultural focus on efficiency.

The lack of effectiveness

The longest time of the last century, a predominant focus on efficiency was okay. Due to the rise of industrialization, the effects of two world wars and some more factors, demand was a lot higher than supply for most of the century. Even when supply started to exceed demand due to ongoing improvements of efficiency and the effects of the globalization, the concept still held because refined marketing and advertisement combined with increasing average wealth created enough artificial demand to persist with a focus on efficiency.

Most of the last century, software development was also defined by a demand that was a lot higher than supply. Additionally, the tasks solved with IT usually were complicated but not complex until the second half of the 1980s. Hence, aiming for more efficiency was fine.

It was only in the late 1980s when software development started to cross the line between complicated and complex due to increasing compute power (and thus the kinds of tasks, IT became able to support), advances in networking, and PCs including their new types of software (like, e.g., spreadsheets) entering the business departments. Still, the focus on efficiency was maintained for a long time – first by developing bigger and bigger software development processes, later by (mis-)using agile methods (usually Scrum) to improve “velocity”, i.e., efficiency.

Fast forward to today:

We still find a widespread focus on efficiency even if the world has changed massively. Today, most markets are post-industrial. This means supply is significantly bigger than demand. As a consequence, sales are no longer easy or even guaranteed. The customers only buy your products or services if they meet their personal needs and demands better than the products of your (usually numerous) competitors. Advertisement and marketing are not able anymore to cover the effects of this development. This creates a high degree of uncertainty regarding the shape, parts and properties of your products and services. Effectiveness is not a given anymore.

Software development has long crossed the line to complexity. In a complex environment, it is not possible to determine upfront what to do when and how much effort it takes. You may have a target but the way towards the target is not predictable anymore. Your actions affect your environment in non-linear ways and vice versa. Hence, you must continuously observe the whole environment and adjust your course based on your observations. Effectiveness is not a given anymore.

Additionally, due to the ongoing digital transformation, IT has become inseparable from business. Business is IT. IT is business. All drivers affecting the business side of a company also affect the IT side of a company. This means, software development is also affected by the shift towards post-industrial markets. It is not clear anymore if requirements will create value or not. Effectiveness is not a given anymore.

Tackling a lack of effectiveness with more efficiency

Our biggest challenge in IT in general and software development in particular is a lack of effectiveness, not a lack of efficiency. Software development, especially the coding part is extremely efficient. Our problem is not that we do not create enough software.

We create way more software than we need. But most software created is just waste.

Most features implemented are never used or only rarely used (see, e.g., this talk summary by Martin Fowler or Randy Shoup’s presentation “Agility and Architecture Together – Enabling a Fast Flow of Change”, starting at 3m33s)

Depending on the sources you look at ³, only 20% to 33% of all features implemented create value. All other features either do not create value or even destroy value. Note that these numbers (between 20% and 33%) come from places that tend to have excellent product managers who already think in terms of uncertainty. Those product managers already act accordingly and split up their features in small series of “bets” or “questions” against the market, making sure they abandon the development of a feature early if the market does not respond to it.

Most “normal” companies do not have such product managers. They implement features in an all-or-nothing fashion. Additionally, they do not recognize the complexity and uncertainty of the market. They still (implicitly) believe that all features required will create value – at least this is how they act: The application does not go live until all features described in the requirements document (or backlog if they are (pseudo-)agile) are implemented.

Of course, they are affected by the uncertainty of the post-industrial markets they live in. They see that their customers do not respond as expected to the features they released. But instead of adopting a hypothesis-driven development approach and slicing a feature into a series of small questions to understand if enough customers care about it (and abandon the feature early if they do not), they simply push for more features: If we just spit out enough features fast enough, we increase the probability to (accidentally) meet the needs and demands of our customers. A big shooting-in-the-dark game at its worst, typically reducing the rate of value-increasing features to approximately 10%.

Additionally, the existing processes and procedures often reinforce such an approach: Yearly budget cycles, yearly project portfolio planning, big setup and controlling overhead for the inevitable projects, one-way software development processes not allowing for fast (or at least any) feedback cycles, lots of manual testing and deployment work, lengthy handovers to production, compliance processes implemented in a way that cement the old approaches, and so on – all implicitly assuming everything being done will create value, being perfectly effective by definition, ignoring the uncertainty of post-industrial markets as well as the complexity of today’s software development.

If you are still unsure if focusing on efficiency is the wrong approach, just go to an average company. Most of them worked for more than 30 years to maximize software development cost-efficiency without compromising quality. They launched dozens of initiatives to improve efficiency and quality. They put lots of money and effort into that goal. Their software development should be blazingly fast and defect rates should be close to zero.

Ask them if their software quality is great. Almost always, the answer is no. As them if their software development is fast. Almost always the answer is no. Still, they keep pushing in the same direction even if everyone knows their approach does not deliver the results they expect. This feels a bit like the famous quote “The definition of insanity is doing the same thing over and over again and expecting different results.”, often (falsely ⁴) attributed to Albert Einstein.

To sum it up in other words: Due to their sole focus on efficiency, in most companies more than 80%, sometimes up to 90% of their software implementation efforts are waste, resulting in huge amounts of idle or even value reducing performances.

Becoming more productive by becoming effective

To stress it once more: The problem is not software developers would code too slow. When it comes to coding, most of them are astonishingly fast – sometimes even too fast in their haste to complete feature after feature, creating a surplus of technical debt. Increasing the efficiency of software developers, making them code more in less time will not solve any problem.

But focusing on effectiveness instead of efficiency would solve a lot of problems. Software development of companies could easily become 2, 3, sometimes 5 and more times more productive by focusing on effectiveness, by doing the things that actually create value instead of focusing on doing things more efficiently, no matter if it creates value or not.

This is a huge lever regarding productiveness. This a way bigger lever than squeezing 1%, 2% or even 10% more efficiency out of the established processes – which usually backfires anyway by introducing a bigger loss of productiveness through the backdoor.

In a world full of complexity, uncertainty, unpredictability and ambiguity, it is more important than ever to focus on effectiveness. Not wasting money for things that do not create any value saves you way more money than all hoped for efficiency gains due to low code, AI tools, quantum computing or whatever is the hype of the day.

If we focus on effectiveness, efficiency will follow naturally. After realizing the gains from improving effectiveness we still will find places that could be improved regarding efficiency and it is perfectly fine to do so. The difference is that the primary focus would be on effectiveness and we only improve efficiency after we are sure we are doing the right thing, the thing that actually creates value.

If we are sure, we are doing the right thing, it is perfectly fine to make sure we do the thing right – but only then.

However, this would require recalibrating our minds from local efficiency optimization to global effectiveness improvement, ultimately resulting in a very different way of approaching the whole business, not only IT. A very long way for many companies, I am afraid …