Degrees of uncertainty

After introducing the concept of uncertainty as the unifying main driver of rethinking IT, I discussed the basic approach how to act under uncertainty in the previous post.

In this post, I will discuss that not everything is equally affected by uncertainty and how we need to respond differently to different levels of certainty and uncertainty.

From certainty to uncertainty

As I wrote before: Uncertainty is the unifying driver of rethinking IT and we need to rethink basically everything that we became used to in the last 50+ years in IT – and we often hold dear.

But before I will dive deeper into the rethinking IT in future posts, in this post I want to make clear:

Not everything is uncertain.

Yes, uncertainty is or will become predominant in many places inside and outside of IT and we must be able to respond to it adequately to survive as a company. Yet, we also need to realize:

• Certainty still exists and the approaches that are great to address uncertainty usually do not work very well in environments dominated by certainty.
• Uncertainty comes in many shades. Things can be a tad uncertain, quite a bit uncertain or even massively uncertain.

I wrote about industrial markets, post-industrial markets and the startup working mode before. All of them bear different levels of uncertainty. Still, it is important to understand that these markets and corresponding modes are not mutually exclusive but rather define a spectrum. ¹

I also mentioned in prior posts that sometimes markets are kept inefficient artificially (via regulation, high entry barriers, monopolism, lobbying, etc.). Thus, even if a market generally has post-industrial properties, it can present itself differently sometimes – again in varying degrees.

Additionally, different projects inside the same company can exhibit very different characteristics, just depending on the mission of the project. It is important to understand the degree of uncertainty associated with a project. Otherwise, very likely we will address it in the wrong way.

To illustrate the problem, let us have a brief look at a few example projects:

1. Implement a new legal regulation into the existing systems (e.g., a new customer documentation law in banking).
2. Replace a system that has reached its EOL (end of lifecycle).
3. Extend or improve an established business offering (e.g., a new tariff plan generation in telecommunications).
4. Create a new digital interaction point for an established business offering (e.g., a subcontractor self-service app in logistics).
5. Grow a successfully launched new business offering (e.g., the “lifeworld” initiatives that many insurance companies started lately).
6. Explore a potential new business offering (e.g., what many companies try to do in their “innovation labs” and alike).
7. Create potential new business offerings (e.g., coming up with potential business offerings based on the vision “create a better world for our children”).

All these types of projects could occur in a single company, and there are quite some companies where all those project types actually occur. All those projects face different degrees of uncertainty, ranging from not containing any uncertainty at all (#1) ² to being extremely uncertain in many ways (#7).

Unfortunately, quite often companies have a single software development process, and all software development projects need to adhere to that process. No matter if they use a traditional process framework or some flavor of “Agile”, it is a one-size-fits-all approach for all their projects, no matter what uncertainty properties they exhibit.

Different value maximization approaches

It should be obvious that it does not make sense to address all the listed example projects the same way if you want to achieve a good effort-value ratio.

If you have a completely certain project it does not make any sense to tackle it using an approach that is optimized for addressing uncertainty. Instead, we should focus on maximizing efficiency. We know for many years how to do this best: Extensive planning upfront. Creating a detailed plan. Then executing the plan step by step. A rigorous controlling to detect and readjust unwanted deviations from the plan.

I often surprise agile hardliners with the statement “It is perfectly fine to use waterfall (if the context fits)”. Waterfall ³ is based on the assumption of complete certainty: Everything necessary for planning is known upfront and requirements do not change in the course of the project. In such a setting, waterfall is the most efficient and thus most valuable approach existing.

Still, such a perfectly certain context rarely exists. Most contexts bear more uncertainty. Either requirements remain unknown upfront no matter how hard we try to discover them and can only be detected in the course of the project using internal feedback loops ⁴. Or you can only learn from external feedback loops, collecting the (unpredictable) user responses to your ideas (hypotheses), based on repeatedly deployed versions of your solution ⁵. Depending on your task you may be affected more or less by both types of uncertainty and need to respond to it.

Moving from project example #1 to #7, you need to give up more and more of the certainty-driven, plan-based approach and utilize more and more of the uncertainty-driven approach as I introduced it in the previous post: treating requirements as hypotheses, take small steps and collect external feedback often to learn and adapt early in order to avoid bigger misinvestments.

Software development approaches

If we break down this general observation to concrete software development approaches, we can arrange them along the certainty-uncertainty axis.

Here I use a rather simple and coarse-grained schema to arrange the different approaches. The intention is to illustrate the basic idea, not to create a perfect schema that takes all particularities of all approaches into account. This would require a much more complicated picture that would obscure the underlying idea. Thus, if you should feel like putting an approach more to the left or to the right, that’s okay. Again, it is about the illustration of an idea, not about a perfect arrangement of software development approaches.

On the left hand I start with approaches that work well under certainty. Usually, their focus is on optimizing costs, i.e., to implement a given fixed set of requirements as cost-efficiently as possible. In some other contexts, their focus is on repeatability (which obviously only makes sense under certainty). Typical representatives of this class of approaches are the aforementioned Waterfall model, the V-model or SAFe ⁶.

The enterprise “Agile” approaches accept a bit of uncertainty. As I stated in footnote #5, their typical implementations only focus (more or less) on the problem of unknown requirements, but neglect the uncertainty introduced by dynamic markets ⁷. Typically, they have a strong focus on optimizing costs while allowing a bit of uncertainty regarding requirements. The latter usually is addressed by over-the-time requirements refinement and a few internal feedback loops. Typical representatives of this class of approaches are Scrum or LeSS.

The next class of approaches support quite some uncertainty regarding implementation. The general goal of the project is clear-cut, but there are lots of external unknowns regarding the concrete measures, i.e., features and their implementations, to reach the goal. They must be explored in a sequence of small steps with lots of feedback from the market.

From a software development perspective this means a focus on flow. The goal is to minimize the time from asking customers a question until they can answer it – which is needed to test hypotheses (see the previous post for more details). The time from question to answer includes the time the accompanying software implementation needs to move through the IT value chain.

Hence, minimizing the time from question to answer requires minimizing the time through the IT value chain – which means a focus on flow. Typical representatives of this class of approaches are DevOps or Lean.

While the previous class of approaches tries to achieve short cycle times without compromising quality, there are contexts where fast learning via very fast feedback is king while quality has a subordinate role. Often you find these conditions in startup contexts where finding a product-market fit before the funding runs out is the single most important goal.

In such a context, there is not only a lot of uncertainty regarding the features and their implementations. You even do not know if your general goal is viable. Approaches like Developer Anarchy as described by Fred George that focus on extreme short external feedback loops can be useful in such settings.

Finally, on the very right side of the axis you do not even have a clear goal but are still looking for it. In terms of uncertainty this is the highest level of uncertainty. You usually have a vision like in the example project #7 but everything else is still undefined and thus uncertain. If you are in this place you should abstain from developing software and focus on finding a goal that is worth testing. This is the domain of innovation approaches, not software development approaches.

Picking the right approach

As you can see, the project context changes drastically while moving from the left to the right and obviously there is not an one-size-fits-all software development approach to handle all those contexts. Instead, to address a given project the right way, you need to understand its degree of uncertainty first and then choose a fitting software development approach. I.e., the software development approach follows the project characteristics, not the other way round.

Unfortunately, many companies still try to impose their single one-size-fits-all software development approach to all their projects, no matter if it fits the project characteristics or not. ⁸

The key is to understand where you are in terms of uncertainty and then pick an approach that suits your needs. No approach fits everywhere. Each one has its sweet spots and its weaknesses. Thus, know a set of different software development approaches and where they fit best, be receptive for the project context, especially its level of uncertainty, and then choose the approach that fits best.

Summing up

In this post I discussed that not everything is equally affected by uncertainty and how we need to respond differently to different levels of certainty and uncertainty.

I provided a few project examples to illustrate the range from certainty to uncertainty – from projects that bear no uncertainty at all to projects that bear so much uncertainty that it does not make sense to start implementing software yet. By looking at these examples we saw that certainty and uncertainty are not a strict dichotomy but rather form a spectrum.

Therefore, it does not make sense to go with an one-size-fits-all software development approach as we see in many places. Instead we need to understand the degree of uncertainty of our project and then pick the software development approach that meets our needs best.

There is no one-size-fits-all, but only a fit-for-purpose.

With this post, I would like to finish my initial introduction of the concept of uncertainty. I started with introducing the concept of uncertainty, continued with the basic approach how to act under uncertainty and ended here, extending the picture to a range from certainty to uncertainty with many grades between them.

I will come back to uncertainty and its consequences many more times on the future. But for the moment, I would like to leave it here and hope that these posts gave you some new ideas to ponder …