The microservices fallacy - Part 11

This post complements the discussion of the previous posts with a few general complementing recommendations and concludes this blog post series.

In the previous post we discussed microliths as a second alternative to microservices. Additionally, we we discussed a few typical challenges that people often face after deciding for one of the three alternatives microservices, moduliths or microliths and how to respond to them.

This post complements the findings of the previous posts with several general recommendations.

Microservices are often introduced because people hope they will resolve some problems that caused pain for a long time. Yet, usually microservices do not address the actual root causes of the problems. Instead they just add another layer of complexity to the IT landscape, i.e., become another painful problem over time. To solve the underlying problems, other measures are usually needed.

Furthermore, if microservices are used as architectural style, it is important to accompany them with some additional measures to reduce the probability that they will turn into just another unsolved problem.

The recommendations made here address both aspects: What you need to do to address the underlying problems that caused the original pain and what you need to do to avoid that microservices become your next pain point. There would be a lot to write about both topics. I will focus on five recommendations.

(1) Learn better design

I mentioned it several times before: The way you separate functionality and expose it via interfaces defines the properties of your resulting system substantially. It does not only affect understandability, changeability, evolvability and maintainability of the code base. It also affects portability, availability, performance, team autonomy and more.

But even if functional design is so essential, it is underrated very often. Reading the computer science literature regarding functional design makes you feel like we are stuck in the 1970s. The discussion about good functional design emerged back in the late 1960s, but around the mid 1970s it basically dried up and the general perception never evolved beyond that point.

With Domain-driven Design we currently see a little renaissance of the functional design discussion. Still, an actual pursuit of the details and intricacies of functional design that goes beyond what we already know from the 1970s is rarely seen. Instead, people often just rename their old practices with new terminology and/or use terms like “bounded context” inflationary without really understanding them.

Better functional design would probably solve 99% of the problems that people hope microservices would solve. But learning how to do good design is hard work and there are not any shortcuts or silver bullets ¹. It is so much easier to ride a hype wave and hide behind “everybody does it” – especially if the expected results are not achieved. But it will not solve your problem.

(2) Do better architectural work

It is crucial to understand that architectural work is not just about mixing and matching technology. A lot of the work is about understanding the problem from all perspectives (market, organization, economics, functionality, business model, operations, customer expectations, …), finding alternative solution options and evaluating their trade-offs.

In the end, you should strive to make the life of the people affected by the solution better with your architecture. A level deeper, this means:

• understanding economic and quality needs across the different stakeholder groups,
• balancing them,
• ensuring behavioral correctness at runtime
• while optimizing the overall costs over the lifecycle of the system
• without compromising the wellbeing of the people affected.

This is something completely different than just juggling some technology and frameworks or jumping the current bandwagon.

Again, this is hard work. But without this work you will not be able to come up with a good (functional) design and the problems that you hoped to solve with microservices will persist.

I discuss architectural work in more detail, e.g., in this slide deck. A subset of the ideas contained in the presentation can be found in written for in this blog post.

(3) Learn better requirements engineering

Many of the tight functional dependencies between modules are a result of poor requirements engineering. First of all the requesters do not know your solution design. Hence, they cannot know if their new requirement messes up the whole existing design. They also cannot know if modifying the requirement a bit without changing the outcome would result in a much better solution.

It is not the requester’s job to know. It is our job as their IT counterparts to help them understand the consequences of their demands. It is our job to work with them to find the best requirements with respect to the problem and the solution domain. ²

Also the traditional separation of duties in software development pose a problem. You basically find the software development process split up in isolated tasks that are connected by a virtual assembly line, called the software development process. In such a setting, lots of valuable feedback and details are lost in the process.

Furthermore, the requirements went through several stages – potentially being altered in each stage based and the needs of the stage – before eventually reaching someone who knows the solution design. And typially that person does not have any means to discuss or alter the requirements anymore. When the requirements eventually hit IT, they are usually set in stone.

In agile projects, usually based on Scrum, often you are not better off. While the project itself is “agile” (often merely being an agile cargo cult), the requirements went through the same stages of the organization before hitting the project – again being set in stone at that point in time.

Even if some degrees of freedom are formally left in the project implementation, you often see powerless Product Owners (PO) whose only role is to decouple the IT from the business department. They are nothing but toothless requirements collectors who have to feed the requirements that they are told in the “agile” implementation process.

The PO often also does not know the solution design and thus cannot discuss requirements in the aforementioned way. And the team members are not wanted to talk to the requesters directly. To be fair: This is not the fault of Scrum. Scrum describes the PO role differently. Still, it is often implemented this way.

To come to better requirements you need to break through the decoupling of requester and solution expert. At least you need a feedback loop with the genuine option to change things. Otherwise your requirement requesters, unaware of the consequences of their acting, most likely will force you into a highly coupled design.

Want vs. need

Another problem is that requirements often capture what requesters want, not what they need. Sometimes you are confronted with some ugly requirement, a requirement that would break your whole design (or at least create high coupling between parts). If you take the time to explore why the requester articulated this requirement, you often realize that the requirement has little to do with the actual need.

Very often, requesters do not know how to step back and figure out the root causes of a problem they experience. Instead, they usually come up with a seemingly obvious solution for the problem, based on their current perception – which is a totally understandable behavior. Most of us tend to act this way, especially if you are under pressure.

But if you take the time to understand the root causes of a given problem, you often see that something different is needed. Or at least you will be able to find different solution options that do not break the design.

Again, you must be able to directly interact with the requester to explore odd requirements, detect the actual need and discuss alternative solution options.

Solutions disguised as requirements

You also find requesters, often with some background in IT, who already have an idea how the solution should look like in their opinion. These people often write implementation instructions instead of requirements. They do not describe what they need but how they expect you to implement it.

As these people usually are not design experts and often even do not know the actual solution very well, chances are that their solution ideas are suboptimal and lead to bad designs with high coupling – in the worst case including ugly hacks “to speed up implementation”.

In such a setting it is important to figure out the actual requirement behind the implementation instructions. Often it is not easy to convince those people to let go of their “solution”, but the alternative is a deterioration of the solution structure.

Again, you need access to the requesters to understand what they actually need and discuss different solution options with them.

There are more problems that lead to bad or overly complex requirements but I will leave it here.

The key observations are:

• Good requirements engineering is essential for good designs that do not deteriorate over time. If you are confronted with contradicting, tightly interlocked requirements, you will not be able to come up with a well-structured, nicely maintainable and evolvable design. “Crap in, crap out” is also true in this context.
• You need direct access to requesters to gather, assess and discuss requirements. It is not about rejecting requirements or fighting requesters. It is about helping them to understand the consequences of their demands and support them finding the best overall requirements – also regarding the resulting solution structure. Again, this is everything but simple, but very rewarding if you master it.

(4) Help people understand the intricacies of IT

I already wrote several blog posts about that topic, e.g., here, here and here. The maybe most critical point: Many people still compare software with physical goods. But software has totally different properties. Software especially must be changed continually after its initial completion to preserve its value. This is totally different from most physical goods and basically changes everything regarding the design process: It must not only work now, it also must not compromise future changes.

As long as people do not understand the difference, software will be treated wrong and bad things will happen. Therefore we need to help people outside IT to understand the intricacies of software.

We need to help people to understand that

• we continuously need to change software after its release to preserve its value.
• the actual costs of a change are not the immediate costs but the costs it implies in the future.
• writing software is design, not a shop floor activity.
• the extreme malleability of software is also a curse and we need to use this property carefully.
• …

(5) Simplify!

I wrote a whole blog series of 15 (!) posts about this topic. So, there is a lot to be said about this topic. The core observation is:

• In IT we drown in complexity and every day it becomes worse.
• At the same time IT becomes more and more indispensable in our private and business lifes.

If we combine these two observations, we see that IT becomes more vital and less manageable every day. It feels like a dance on a volcano that is about to explode every moment.

While we cannot avoid all complexity (some is essential, i.e., needed to solve the problem at hand), we often pile up lots of accidental complexity, complexity not needed to solve the given problem. This happens on all levels:

• Not understanding market demands and responding in a wrong way to them
• Bad IT governance processes
• Budgeting and process approval madness
• Project bureaucracy
• Running after technology fashions
• Architecture narcissism
• Implementation detours and vanities
• Piling up archaeological layers of technology over the decades

and more.

Microservices – the topic of this post series – used for the wrong reasons result in more accidental complexity. They are a very demanding and complex architectural style. If we use them even if a lot simpler style would have been perfectly sufficient to address the given problem, we are complexity culprits.

Try not to be a complexity culprit, try to be a simplicity defender.

Conclusion

These were the general recommendations I wanted to add. They may sound simple, maybe even obvious. But do not be fooled: They are all very hard in practice – to learn as well as to overcome the resistance you will face.

Still, I am convinced they are worth aiming for as all of them help to make IT a bit better and more enjoyable place: Problems will become less, things become more cost-efficient, work becomes more fun, …

This was a lot of stuff we discussed in this blog series. Let me just recapitulate a few highlights here:

• The most widespread reasons why you “need” microservices are fallacies. We have debunked them one by one.
• Microservices in themselves do not have any value. If you should need them they come as a natural byproduct of something bigger.
• Distributed systems are hard. They are one of the hardest to understand domains in IT. Try to avoid distribution if you do not really need it.
• You need modularization. You can have it without going for microservices. The actual problem is that good functional module design is still poorly understood.
• Learn better design and requirements engineering. This will help you many times more with your problems (the problems that made you adopt microservices) than microservices could ever do.

And maybe a last important recommendation:

Do not fall for hypes!

There is a multi-billion Euro industry that wants you to fall for hypes because it makes its living from it: Product vendors, cloud service providers, consulting companies, training providers, media, all of them. They all diligently fuel the hypes.

But their acting is not about you. It is about them and their profits only. The more headaches you have (or think you have), the more money they make. It is important for their revenue that you never run out of headaches.

Have that in mind before you run after the next hype you read about, hear about or somebody tries to sell you. ³

Moving on

I hope I could provide you with a few ideas and insights, some stuff to ponder. It is not that I dislike microservices. Quite the opposite: Personally, I love distributed systems. I deal with them for more than 25 years meanwhile and I am still fascinated by their intricacies – must be the engineer in me. Therefore, microservices in their original form are exactly my cup of tea, confronting me with all their intricacies.

Yet, I know about the problems of getting them right and reliable, and the longer I travel IT, the more I realize that most people have a really hard time getting their heads around them. ⁴

Additionally, I realized that most companies not being hyperscalers simply do not need microservices. There are simpler architectural styles that are perfectly sufficient and that are a lot easier to manage, especially if you come from an enterprise software development background. They are just not so fashionable …

Link to the first part of this series containing the TOC