Simplify! – Part 3

In the first post of this blog series, I described two antagonistic observations: IT has become an indispensable part of everyday business and private life while at the same time the complexity on the IT solution side grows all the time, meanwhile being on the verge of unmaintainability.

In the previous post, I discussed that the excessive complexity is extremely unhealthy for the people affected as well as for the companies they are working for. Additionally, I described that the “survival” patterns that the people apply to escape the stress induced by the complexity create a self-reinforcing loop that in the end makes things worse instead of creating a relief – especially if combined with constant efficiency pressure as it is normal for most companies.

In this post, I will discuss the difference between essential and accidental complexity as it is required to understand this difference for the rest of this blog series:

• On the one hand, it is not possible to get rid of the essential complexity needed to solve the problem at hand, i.e., it also becomes counterproductive if you try to oversimplify the solution.
• On the other hand it is important to understand that most of the complexity in IT today tends to be accidental, i.e., is not needed to solve the problem at hand. This is what we need to get rid of.

But to be able to do so, it is first necessary to be able to distinguish essential and accidental complexity – thus this detour.

Essential vs. accidental complexity

Essential and accidental complexity ¹ are very fundamental concepts not only in IT. Unfortunately they are forgotten often.

In the domain of computer science, essential and accidental complexity became popular with “No silver bullet” ², a famous essay that Fred Brooks published in 1986 ³. In his essay, Fred Brooks started:

Following Aristotle, I divide [the difficulties in software technology] into essence – the difficulties inherent in the nature of the software − and accidents − those difficulties that today attend its production but that are not inherent.

Then he described four properties that in his opinion lead to essential complexity in software development, i.e., that are not avoidable. I try to summarize them briefly (for the full description, please read the essay):

• Complexity - This first property basically is a collection of drivers: The multitude of different parts, the lack of standardized parts (I already touched that topic in my first post about the reusability fallacy), the non-linear growth of complexity if adding new parts, the problem that complexity cannot simply be abstracted away (like in mathematic models) if you want a working solution, the problem of unwanted and unexpected side effects and more.
• Conformity - This is basically the extension of the first property to the interaction of parts: The absence of a unifying principle in the interaction of software parts (on a functional and technical level), leading to added complexity when trying to make different parts work together. Basically, as I understand it, it is about the complexity needed to match heterogeneous concepts on a functional and technical level.
• Changeability - Software is one of the few (if not the only) materials where a product crafted from it is not final after its completion and release. While in manufacturing, things tend to be very infrequently changed, once they are completed, in software initial completion usually is just the beginning. Software is always subject to continuous change until the end of its lifecycle. Additionally, successful software tends to outlive its environment, i.e., needs to be ported to new environments, which also requires change.
• Invisibility - Software is invisible and lacks good visualization tools that help to understand if a requirement makes sense, contradicts given constraints or lacks relevant aspects. E.g., it is easy to figure out upfront that it makes little sense trying to put a cabinet that is 3,5m wide in a room that is 2m x 3m. In software we lack such clear and simple abstractions and tooling which makes reasoning about software inherently hard.

Everything resulting from these four properties of software, Fred Brooks concluded, is essential, non-avoidable complexity, while all other, additional complexity we face in software development is avoidable.

Shortcomings of the “No silver bullet” definition

To be frank, I am not a big fan of Brooks’ definition of essential complexity. I only listed it here because the paper and thus the definition Brooks gave are so famous that you cannot discuss essential vs. accidental complexity without mentioning this definition.

Personally, I think that the four properties are not the core of essential complexity in software development. Each of the properties contains valuable ideas that are absolutely worth pondering.

Yet, I think the properties mentioned do not describe essential complexity, but drivers of accidental complexity instead:

• Neglecting the facts that software lacks standardized parts (as manufacturing has) and complexity cannot simply be abstracted away to come to a solution (as it is possible in math), and then applying concepts from manufacturing or math on software developments leads to a lot of accidental complexity.
• Ignoring the need for integration with other software parts when designing a piece of software leads to different concepts and interfaces for each part of a software solution, leading to a lot of accidental complexity.
• Ignoring that a software solution is under continuous change pressure until it is retired and thus applying traditional project management techniques to it that neglect this fact, we add a lot of accidental complexity.
• Not being able to understand if a requirement contradicts the existing solution, does not fit in, lacks vital aspects or alike because the material “software” is invisible and lacks good visualizations that would uncover such shortcomings, we add a lot of accidental complexity.

Therefore, while I think that Fred Brooks made several important observations in his essay that all are relevant for the discussion of complexity, at least from today’s perspective I think his four properties are rather drivers of accidental complexity than a description of essential complexity.

A better definition

I think that essential complexity is defined in the problem domain, i.e., the task to be solved defines how much complexity is needed in the solution domain, the software solution.

Thus, the functional requirements, the non-functional requirements and the mandatory constraints (organizational, economical, legal, technical, …) define the essential complexity. Everything we do that is not needed to create a solution for the given requirements within the given constraints is accidental complexity.

Consequently, I prefer the definition that Moseley and Marks provide in their paper “Out of the Tar Pit”. They define essential and accidental complexity like this:

Essential Complexity is inherent in, and the essence of, the problem (as seen by the users).

Accidental Complexity is all the rest —- complexity with which the development team would not have to deal in the ideal world.

While probably also not being perfect this is the best definition, I have encountered so far. Even if it leaves quite some room for interpretation, I think it is precise enough to distinguish essential and accidental complexity quite well in most situations.

Especially, the addendum “as seen by the users” is vital, as we will see in the upcoming posts of this blog series. Thus, I tend to use this definition if I try to describe essential and accidental complexity.

Taking a closer look

Taking that definition of essential and accidental complexity plus my additional notes, we could illustrate the concept of essential and accidental complexity like this:

On the one hand we have the problem complexity. It is defined in the problem domain: the functional and non-functional requirements plus the required constraints, like legal, commercial, temporal and other business-related constraints.

On the other hand we have the solution complexity. It is defined in the solution domain: the concepts, implementation, tools and technology used plus IT-organizational and technical constraints.

The problem complexity defines the essential complexity. The solution complexity cannot be lower than the problem complexity. Otherwise, the solution would not solve the problem. Ideally the solution complexity equals the problem complexity. In that case, we do not have any accidental complexity.

If the solution is more complex than needed to solve the given problem, i.e., if the solution complexity exceeds the problem complexity, the excess complexity is accidental, unneeded complexity.

Easy, isn’t it. If we make the solution more complex than the business department required us to do, we added accidental complexity. Hence, as soon as we stop to implement more complex solutions than our business peers require us to do, the accidental complexity will disappear.

Sounds too easy? Well, yes, it is. It is not that simple. If we take a closer look we can see that accidentally complexity comes in several different flavors and can be found at different places, and not only on the solution side.

Summing up

In the first version of this post, here followed the closer look. Unfortunately, this resulted in a post that was way too long ⁴. Therefore I decided to split it up in two parts and discuss the different types of accidental complexity in the next post.

In this post we have discussed the difference between essential and accidental complexity. While we cannot get around the essential complexity of a given problem (at least if we want to solve it properly), accidental complexity makes things just harder without adding any value to the solution. Therefore we need to tackle it relentlessly if we do not want to fall off the complexity cliff in IT.

In the next post we will dive deeper in the different flavors of accidental complexity. A little teaser: we will see that it also hides in usually overlooked places. So, stay tuned …