Simplify! – Part 7

In the previous post of this series I discussed some drivers on the company level that (among other things) cause a lot of accidental complexity in IT. The drivers were not understanding IT and its role today.

Together with not understanding post-industrial markets and digital transformation which I discussed before, they ultimately form the root cause for many other effects and evolutions, we will discuss in this and the subsequent posts.

In this post, we will begin with some of the effects that we see in and around IT projects. They topics we will address are:

• Counter-productive budgeting processes, treating IT as a cost center
• Useless requirements due to a wrong focus
• “Agile” and “DevOps” cargo cults, not improving anything
• The “high utilization means more productivity” fallacy
• Enterprise Architecture Management as an impediment
• Compliance/governance implemented wrong adding accidental complexity

In this post I will start with the first two topics. In the next post I will continue with the other topics. ¹

The accidental complexity we will discuss in this post will all be of the first kind – unnecessary requirements.

The fight for the yearly budget

A big source of unnecessary requirements are the project budgeting and approval processes that are still predominant in many companies. It usually starts on the company level. The different department budgets are allocated. As legal requirements demand annual accounts and all figures need to be clearly allocated to a fiscal year, the budgets are usually also allocated in a yearly manner.

Also the budgeting process tends to be so complicated and exhausting that nobody wants to endure that procedure more than once a year. As the demands for budget are always higher than the available budget, everybody fights to get their share, at the expense of the others. This leads to long, tedious application and review processes where every party needs to explain in great detail why they think they need the money more than the others.

Over time those processes became a yearly ritual where all parties continuously upgraded and perfected their weaponry. This means that you usually claim way more budget than you need because everybody does it and you hope that after the usual cutback procedure you will end up with what you actually wanted. To justify the surplus on budget demand you create huge bloated initiatives, projects and alike, expecting that they will be cut down in the review and negotiation process.

But more often than not, things are not cut down in the expected way. Often the bloated initiatives and projects either survive or get killed as a whole. This is way easier and a lot less work than going through the details of all the different applications and rightsize them in a sensible way.

After the negotiation circus, following strict rituals, is over and the dust settles, some initiatives and projects survived – in their whole bloated splendor. This means, each year starts with some projects that are a lot bigger than they need to be, just because it is part of the yearly budgeting ritual to ask for more than you need.

Now would be the time either to rightsize the project and use the freed budget either for other sensible projects or not to use it at all. But this is not possible because it is part of the rules of the game that you need to use the budget exactly for what it was allocated for. And if you use less budget than you claimed, your budget will be cut by default the next year.

Additionally, all the other parties who did not get your share of the budget enviously watch your project, ready to use any perceived deviation from the budget application against you to get a bigger share of budget themselves in the next year.

In the end, as no one want to stand back in the yearly budgeting process, we tend to end up with bloated projects that will be implemented as described in the application, no matter if it is actually needed or not.

It is the result of a game, all departments play – even if everybody knows that the game does not make sense. As long as not all parties agree at the same time to change the game, it is unlikely that the game will ever change:

Culture and habit trumps meaning and company wellbeing.

On a side note, this also means that:

• budget claims will never become smaller, only bigger as the upgrading game continues
• allocated budgets will always mean a lower bound, i.e., nobody will spend less than the allocated budget

Of course, as nobody likes the process and everybody knows that it does not make sense the way it is, there are regular attempts to improve it, to reduce the potential for misuse and make the process “fairer” or alike. But typically those attempts end with even more rules, more work to prepare the applications, more reporting – and a slightly changed, maybe a bit more subtle, but definitely more complex version of the same game.

Admittedly, this is only a very coarse-grained description of what is happening in many companies. Many would contradict and refer to the intra-yearly renegotiation processes and alike. So, in practice you will probably find more complex variants of budgeting processes than I described here.

Still, most of them tend to follow the overarching ideas and patterns I described, being sort of an arms race that everyone dislikes and nobody knows how to escape. It is what Donella Meadows calls the “escalation” system trap in her highly recommendable introduction to “Thinking in Systems”. ²

Repeating pattern on project level

IT also participates in the game. But usually IT, being considered a mere cost center, only goes for operations and maintenance budgets, i.e., keep things up and running. The “innovation” budgets, i.e., the actual development project budgets have to come from the business departments ³. As a result, the same “fighting for a limited resource” pattern repeats a level deeper when it comes to implementing the budgeted projects.

Most of the projects require IT support, i.e., a software development project or alike needs to be set up which requires support from the IT department. Coming from a strictly industrial thinking that puts highest priority on cost efficiency, plus the fact that IT is considered a cost center, the capacities that the IT departments can provide typically do not suffice to meet all the demands. As a result, the same fight over a limited resource (here: IT support capacity) repeats in project portfolio planning. ⁴

Every party with a budget will try to get their project staffed. If you do not get the IT capacity you need, you will have to wait until the next planning session starts and try again. Therefore, usually project scope and importance tends to be bloated even more to win the race for capacity.

If you win the race, you will experience an amplifying effect: getting the capacity you need gives you a disadvantage in the next planning session. You got capacity now, so it is the turn of other parties next time. This means, as a department you know that you usually have one shot before you have to wait until everyone else had their shot.

Therefore, you try to get most from your shot. You put everything that comes to your mind in that shot, in that project – again, this is the only shot that you have for a year or two, or even longer.

This means that the poor persons who are put in charge of the requirements for the project will list everything that comes to their minds, that they hear, that were told or just can imagine. You only have that one shot for a while, thus better put too much into it than potentially missing anything that might become useful in the future.

The result: Even more bloated requirements.

Also this cycle tends to be an escalation system trap as multiple parties fight for a limited resource. So, everyone tries harder each year to get hold of the scarce resource and everyone tries to squeeze the last bit out of the resource once they got hold of it.

Inward-bound focus

A last driver of valueless requirements is the inward-bound focus that we often experience in organizations that exist for a while. I already mentioned this effect in my posts discussing the shortcomings of business processes and uncertainty.

A big driver of this effect can be explained with the concept of autopoiesis that Niklas Luhman used in his system theory to describe the self-creating and self-preserving properties of systems (like a company or its parts).

In short and extremely simplified: Systems form to fulfill a purpose. The purpose is defined externally – some need, demand or necessity. Over time the systems decouple from their original purpose and become an end in itself. Especially, their whole evolution is then driven by internal needs instead of external drivers (Luhman calls such systems “operationally closed”). I will discuss this topic in more detail in a future blog post.

At the moment it is sufficient to understand that most companies and their organizational parts tend to decouple themselves from their external customers and their needs and eventually revolve around themselves. They become an end in itself. All activities, all changes ultimately are driven by internal needs and demands, not external ones.

This also means that many business departments tend to define new products and services (or changes in existing ones) completely without incorporating their customers. You might have heard sentences like “No, it is not necessary to ask the customers. We know better.” in some of your projects. I heard it in quite some projects when I pressed to collect customer feedback early.

This inward-bound focus that most organizations develop after a period of time results in more requirements that create no value, i.e., that the customers either do not care about or even dislike. ⁵

Neglecting uncertainty

While all of the aforementioned issues are bad enough, they probably are not the biggest problem. The biggest problem is that the whole budgeting process, the project staffing and the inward-bound focus all completely neglect uncertainty. They are all based on the assumption of complete certainty. All this acting are based on the belief that it is possible to reliably know upfront which activities will create value in the future.

Usually there is at least a year, often several years, between the time when the budget application is prepared and the time when the results will become visible. In a highly dynamic post-industrial market, the likelihood that the underlying predictions the budget application was based on will still be accurate at the time of release is about zero. If you add the inward-bound focus, the likelihood becomes even smaller.

This means:

When requirements hit the implementation projects, their value proposition is mostly random.

In 2002, Jim Johnson, chairman of the Standish Group, gave a keynote at the XP conference. There he presented two studies that explored how many features of a software product are needed and actually are used often. Based on the two studies, less than 25% of all features satisfied that criterion (see this post for a few more details).

That observation reinforces the former statement quite well. The majority of features that are usually implemented in a software product create little or no value. Not seldom they even destroy value. The time gap between the moment when someone came up with an idea and the moment when the customers experience the result is way too long – even in a just mildly dynamic market.

Improving the situation

We face IT projects that often start with bloated requirements due to internal budget and resource allocation rituals and where the viability of requirements is not challenged via customer feedback before the project ends. This makes up a big part of the first source of accidental complexity:

• Unnecessary requirements not adding any value to the solution

Again, the question is: What can we do to improve the situation?

As in the previous posts discussing accidental complexity at the market level and the company level, probably the most sustainable relief is to help people understand – especially uncertainty and its effects.

If you think that I harp on about uncertainty: based on what I see every day, certainty-based thinking and acting is still extremely predominant in most companies, resulting in so many counter-productive practices which (not only) lead to a lot of accidental complexity. Hence, it is essential to discuss uncertainty whenever an opportunity is afforded.

Beyond Budgeting and BetaCodex

A great source of support in this context are Beyond Budgeting and BetaCodex. Beyond Budgeting started in the late 1990s. It basically rethought management and governance, understanding that in highly dynamic markets the traditional practices, driven by an industrial mindset, are not particularly useful anymore. Even if the movement is called “Beyond Budgeting”, it is not limited to budgeting but covers holistically how to govern a company. ⁶

BetaCodex arose from Beyond Budgeting, being driven by people who focus on transformation. Consequently, its initial name was “Beyond Budgeting Transformation Network”, but eventually they renamed their movement to “BetaCodex Network”.

While there are some differences, the core ideas of both approaches is the same, simplified in my words: How to run a company that is exposed to uncertainty due to dynamic, post-industrial markets.

Therefore, the concepts, frameworks and principles from Beyond Budgeting and BetaCodex are highly valuable to address change. Especially BetaCodex with its strong focus on transformation provides a lot of advice how to drive change. Additionally, both approaches give you a lot of literature and success stories to back your ideas with (you will need this backing to counter the to be expected resistance).

Hypothesis driven development

On a more concrete level, hypothesis driven development is a very powerful tool as it makes uncertainty explicit. Additionally, requirements will not be implemented just “because they exist”. Instead, they are parsed in a series of smaller questions against the market to validate the actual viability of the original requirement.

This market feedback loop does not only uncover many bloated, valueless requirements that were created due to the effects of the budgeting and project staffing processes. It also debunks many valueless requirements that were created due to an inward-bound focus and neglecting uncertainty.

Of course, it takes energy to convince people with a certainty-based mindset to start thinking in hypotheses. But in the end, this measure is easier to explain and test than, e.g., understanding and counteracting the general effects of uncertainty. While the latter has a bigger lever, it often feels too abstract for many of the affected people, i.e., they have a hard time connecting the insights with their daily work.

Hypothesis driven development on the other hand can be immediately connected with the daily work. And if people do it for a while, it will also start to change their way of thinking, i.e., very likely quietly they will start to embrace uncertainty.

Theoretically, agile software development would support hypothesis driven development out of the box. Unfortunately, most implementations do not. I will discuss this topic in more detail in the next post.

Summing up

In this post, we started to discuss some of the effects that we see in and around IT projects leading to excessive, unneeded complexity. We discussed the effects of traditional budgeting processes, traditional project staffing processes and inward-bound thinking, all neglecting market dynamics and uncertainty.

We also discussed some potential counter-measures that help to mitigate the negative effects of the aforementioned bad practices. As always, there are certainly a lot more issues that would be worth discussing and also more useful counter-measures.

E.g., I often see companies spending huge amounts of money in creating highly sophisticated marketing measures, including AI-driven recommendation engines and more, just to lure customers into buying more mediocre products that otherwise would not sell. In the end the company maybe sells a few percent more products.

Why not spending just 10% of the time and budget wasted for all those marketings measures into developing a really good product the customers really want (because it creates value for them) and then selling 50% more products?

That way of thinking could reduce accidental complexity by orders of magnitude. Yet, discussing the advantages and challenges of this quite different way of approaching the situation would make the already too long post even longer. Thus, I will leave it here, just pointing out that more alternative approaches are available.

In the next post, we will continue the discussion of accidental complexity on the project level due to bad constraints. So, stay tuned …