The essence of architectural work - Part 8

In the previous post, we discussed (decision) uncertainty as a key factor that determines how well a given approach works regarding the When and How much of architectural work.

In this post, we will apply this knowledge and align architectural work with the degree of uncertainty.

Agile and what uncertainty is not

But before we dive in, let us first get rid of some widespread misunderstandings.

We sometimes find reasoning like this:

• The agile principle “Inspect & Adapt” is about navigating under uncertainty.
• We are agile (Scrum or SAFe).
• All problems solved.

Unfortunately, it is not that easy. The point is that we find several issues that all tend to be filed under “decision uncertainty”:

1. The first one is accidental uncertainty due to homework not done. It is unclear what the system does, and thus it is unclear what exactly needs to be changed. Nobody knows how the systems interact. Database structures are undocumented. And so on. This is not actual uncertainty. These are avoidable problems caused by a lack of diligence that can be solved conclusively by doing the required work (and applying the Boy Scout rule ¹).
2. The second one is semi-accidental uncertainty due to missing internal feedback. There is no back channel to discuss requirements that work against the existing system ². Questions are not answered in a timely manner. Relevant information is held back due to political games. And so on. I call it “semi-accidental” as this is rooted in deficiencies of a company’s organization and culture. We would not face these problems with a more suitable company organization and a better culture. Still, in most organizations, these problems persist, and it can be challenging to overcome them. Just doing the work is usually not sufficient. It often takes political finesse and perseverance to address these issues successfully. Still, even if they are harder to solve, they are not actual uncertainty but homemade problems.
3. The last one is uncertainty due to missing external feedback. This is the only non-avoidable type of uncertainty as it cannot be solved within a company. This is the essential decision uncertainty caused by VUCA, which must be addressed via feedback-based approaches.

If we look at the usual enterprise “Agile” implementations, no matter if Scrum, SAFe, or some homemade “Agile” approach, we can observe the following:

• The first kind of uncertainty is often reinforced by enterprise Agile due to deteriorating documentation, including essential documentation.
• The second kind of uncertainty is what enterprise Agile typically tries to address.
• The third kind of uncertainty is usually ignored by enterprise Agile.

Originally, agility tried to address essential uncertainty, the uncertainty due to missing external (market) feedback, the decision uncertainty resulting from VUCA. Unfortunately, 95%+ of all “Agile” implementations in enterprises at best address their internal deficiencies, but completely ignore external forces. Due to cargo cult-type implementations, they often even cause accidental uncertainty by not addressing simple and complicated tasks adequately. Based on its original ideas, agile could perfectly address uncertainty, but in practice, it rarely does.

To wrap this section up:

• Essential uncertainty is based on external factors, i.e., factors we cannot influence within a company.
• Uncertainty based on internal factors is not essential. It is solvable even if culture, habits, and other impeding factors make it hard.
• Enterprise “Agile” does not address essential uncertainty, even if the original ideas of agility did.

Please keep this sometimes sobering observation regarding enterprise “Agile” in mind. It will become relevant when we organize different software development approaches along an uncertainty axis later in this post. ³

The uncertainty continuum

After this excursion regarding Agile and what uncertainty is not, let us return to the main discussion thread. We discussed how uncertainty influences our approach to architectural work, and how we need to adapt our approach to the degree of uncertainty. Architectural work always needs to be aligned with the encompassing software development approach, which in turn needs to be aligned with the degree of uncertainty.

The ultimate goal is to create value with our work. However, as the means to (likely) create value are very different depending on the degree of essential uncertainty, we need to align our software development approach with the degree of encompassing uncertainty.

As shown in the illustration above, we basically have a sort of slider going from full certainty to full uncertainty. On the full certainty side, find:

• Extensive effort planning – Detailed upfront task breakdown and implementation planning
• Maximize efficiency – Focus on cost minimization
• Detailed controlling – Extensive cost controlling to track and correct deviations from the plan

As everything can be planned fully upfront and all potential implementation uncertainties can be removed upfront, the optimization criterion is to maximize efficiency, i.e., minimize the costs needed to implement the given task.

At the full uncertainty side, we find:

• Hypotheses validation – Validate ideas during the implementation to resolve uncertainty gradually
• Small steps – Minimize risk of misinvestments in ideas that prove to be ineffective
• Quick feedback cycles – Keep risky investments small and minimize time until detection of ineffective ideas

As we do not know upfront which of our solution ideas will be effective, i.e., create the desired value, we cautiously need to approach the value-creating solutions. We create hypotheses, break them down in small steps, and validate – or invalidate – them along the way. This way, we minimize the risk of spending lots of effort and money on ineffective solution ideas.

In practice, we rarely face the two extremes, i.e., full certainty or full uncertainty. We rather find ourselves somewhere between the two extremes, depending on the task and context. The more we move to the certainty side, the more plan-based we can work. The more we move to the uncertainty side, the more hypothesis-driven we need to be.

This results in different approaches to maximize the likelihood of creating value depending on the amount of decision uncertainty we are confronted with. In other words, there is no “one-size-fits-all”, but rather a continuum of approaches depending on the level of uncertainty, and it is up to us to pick the one that fits our task and context.

The uncertainty continuum applied

However, it would be a lot of effort to define a custom approach for each new project based on its specific goals, context, and constraints. Instead, we organize them in a few more general software development approaches that together cover the spectrum from certainty to uncertainty.

In the image above, we see several widespread software development approaches arranged from certainty at the left end to uncertainty at the right end:

• Focus on cost – At the certainty end of the axis, the core focus of the approaches is cost efficiency. As the underlying assumption is that very little uncertainty exists, the focus makes sense. Well-known examples of this category are Waterfall, V-model, RUP (Rational Unified Process), or SAFe. ⁴
• Enterprise “Agile” – The usual “Agile” approaches we find in enterprises focus on semi-accidental uncertainty due to internal dysfunctions, especially missing feedback. But at least some of them, like Scrum or LeSS, can also handle a certain degree of external uncertainty. SAFe is not listed in this section because due to its extreme division of labor and long communication chains, it stifles effective handling of market feedback.
• Focus on flow – The middle ground between full certainty and full uncertainty is covered by approaches that focus on flow. They make sure that lead times do not become too long and that feedback, including external feedback, can flow effectively through the system. Typical examples are DevOps, Lean, and Kanban.
• Focus on feedback – When uncertainty starts to dominate the play, fast and immediate external feedback becomes crucial. A few approaches exist that focus on rigorously minimizing feedback time, like, e.g., Developer Anarchy, a method promoted by Fred George (see, e.g., this presentation by Fred George introducing the topic or this written introduction and analysis of it).
• Focus on innovation – Finally, at the very right side of the slider, there is only uncertainty. This is the realm we typically live in when it comes to real innovation, going where no paths exist yet. However, besides a few spikes and prototypes to validate an idea, this is not the domain of software development. A few uncertainties should have been removed before it makes sense to develop a software-based solution.

As I discussed before, the typical “Agile” implementations as we see in most companies are just a bit right of total certainty as they fail to address external uncertainty. But as I also discussed before: Even if the enterprise “Agile” approaches fail to address external uncertainty, they can be useful to mitigate internal deficiencies.

It is this spectrum of software development approaches that I tend to call “post-agile”: opposed to the conviction of many agile advocates, there is not a single “right way” of developing software. There are many ways – even more than shown here, and with many more nuances that fit into this post. We always need to look at the level of VUCA and decision uncertainty that affects us in our given situation, and then decide how to address the situation appropriately. Sometimes, taking a plan-based approach is fine. Sometimes, a bit of “Agile” does the trick. Sometimes, we optimize for flow, or exploration. And sometimes, it simply is too early for software.

The place of architectural work

As I wrote before, architectural work needs to be embedded in the encompassing software development approach. It does not make much sense to discuss architectural work without looking at the surrounding development approach. We cannot explore the architecture over time if the development approach assumes that all decisions are made before the first line of code is written. It also does not make any sense to define the architecture fully upfront if we use a flow-based approach that assumes that we step-wise approach the right solution over time – business-wise and technologically.

Therefore, we needed to discuss software development approaches along the certainty-uncertainty axis first. Understanding the relation between (un)certainty and software development approaches not only helps to overcome the one-size-fits-all trap, and to pick a software development approach that actually fits the task and context. It is also a prerequisite to organizing the When and How much of architectural work, embedded in the respective approach.

Interlude

In this post, we first clarified what we mean if we talk about uncertainty and got rid of a few common misunderstandings. We also discussed how “Agile” usually does not help to tackle uncertainty, even if the original ideas did ⁵. Then, we defined the certainty-uncertainty continuum. Finally, we placed widespread software development approaches along the continuum the axis spans.

In the next post (link will follow), we will use some project examples to clarify further which approach to use best for what type of task. Then, we will add the solution lifetime axis and its effects to lay out finally what to do When and How much to do regarding architectural work. Stay tuned … ;)