Software - It’s not what you think it is - Part 7

In the previous post, we summed up the misconceptions we discussed in this series and what they mean for the current discussion if AI solutions will replace software developers.

In this final post of this blog series, we will discuss what the misconceptions mean for the humans affected by them and how we could possibly improve the situation. Let us dive in.

Software is a unique material

The first observation, we can make when looking at the misconceptions is that software is a quite unique material. Many problems in software development arise from the fact that people try to compare software to physical goods. We see it in the assembly line fallacy. We see it in the value preservation dilemma. We see it in the malleability curse. And the fact that software is invisible and thus deprives us from the reasoning instruments we have in the physical world makes it even easier to fall into the trap.

By regularly comparing software to a physical good, we deprive us from the unique properties software offers:

• It is extremely malleable. We can turn it into almost everything we can imagine.
• It can be modified easily after its initial construction.
• We can build only parts of the final product and still create value for the users (try this, e.g., with a car).
• We can explore how the product should look like in a continuous feedback loop with the users.
• We can easily work with multiple versions of the same product at the same time.
• Building the product is unbelievable fast and cheap.
• Creating multiple instances of the product is even faster and cheaper.

Software gives us options, no other material gives us. However, it also comes with quite unique constraints:

• Software lacks a visible evaluation system. There is no easy way to check if the product makes sense or not by simply looking at it, touching it or alike.
• It must be modified continuously to preserve its value. If external demands change, you do not create a new product. Instead, you modify the existing one.
• A software product tends to live very long and needs to be modified in unpredictable ways over its lifetime.
• Software products tend to become very complex (or rather extremely complicated) over time. It requires explicit effort to avoid its deterioration due to increasing entropy. ¹

Doing it wrong

If we neglect the unique properties and constraints of software, we end up in a place that is very undesirable for the people affected by it. We build up practices, habits and rituals that neither take the unique strengths nor the constraints of software into account. In the worst case, we work against the strengths of software and turn the constraints into weaknesses:

• We try to organize software development as an industrial assembly line, a shop floor process, depriving the people involved from the collaboration means required to do the design work (remember: all software development is design in the traditional sense). It also deprives people from the possibility to continuously learn from the customers by using quick feedback loops for experimentation and learning.
• We organize software development as projects which by their very definition neglect everything outside the project scope and after the project end date. Projects are successful if they deliver their predefined scope within time and budget. Projects are not evaluated by what happens outside their scope or what happens after the end of the project – and it shows. Software lives a long time after the project end and needs to be changed continuously. If it is only written with delivering in time and budget in mind, its long-term changeability will suffer because it is not a project goal ². Software systems also tend to interact in complex ways. If we only consider the parts inside the scope of the project, the interactions with the parts outside the project scope will merely be an afterthought, increasing accidental complexity.
• Additionally, projects tend to have significant fixed costs, efforts and risks. Therefore, usually a huge project ceremony accompanies it, starting with feasibility assessments, budgeting theater, project portfolio planning, lots of reporting and controlling, steering committees, and so on. This does not only result in very long lead times. It also leads to large projects and deprives us from the option to learn anything along the way because changing the plan would mean going through the whole process again.
• We build the whole solution at once in a waterfall-ish way, based on a complete requirements document or an (also complete) backlog instead of experimenting and learning from customer feedback along the way, depriving us from the option to build a product that is much better aligned with the needs and demands of the customers than the initial version. This is also true for most “Agile” approaches. The requirements are also settled upfront and are rarely changed except for adding more requirements along the way. And the feedback loops usually never reach the actual users. In the best case, some internal communication mishaps are clarified. But usually, “product demos” are only treated like another type of status report.
• We are so obsessed with the assembly line fallacy that we forget to automate our actual assembly line, the building, testing and deployment of our systems and underlying infrastructure. Instead of automating all those repetitive tasks, we leave them manual, making our actual assembly line slow and error-prone.
• Because software is invisible and we still confuse it with physical products, we neglect its internal structure and future viability and focus only on the current business features it exposes. We neglect the fact that software needs to be changed continuously to preserve its value and that its extreme malleability can quickly turn against us. We forget to manage it properly, resulting in brittle, error-prone solutions that take longer and longer to change and evolve.
• And so on …

As a consequence, we end up with:

• Long lead times
• Brittle solutions
• High error rates
• Hard to modify solutions
• Exploding (accidental) complexity
• And so on …

Sometimes, it feels as if we have one of the greatest materials at hand but because we do not understand it and thus misuse it, we make the worst out of it.

The effects on humans

Such a setting is extremely frustrating for all people involved and affected:

• The customers are unhappy because they do not get a product that fits their needs and demands. If they are real customers, they will probably turn towards a competitor and use their product (unless you are lucky and their software development processes are as bad as yours). If they are user who do not have a choice, e.g., if they are users from your business departments, they will be really unhappy because the product does not only not fit their needs and demands. It also takes very long to change anything, resulting in frustrated employees that are by far not as productive as they would be if the solutions would fit their needs.
• The business experts are unhappy because it takes so long until their requirements become reality. It also always costs more than expected and takes longer as expected – at least based on what they expect.
• The software developers are unhappy because they are forced to juggle with crappy code that is brittle and hard to change. Additionally, they are continuously accused to be slow, sloppy, unreliable and everyone pushes to make them more “efficient” (or replace them with AI), no matter what wonders they work based on the dreadful state of the existing code and the requirements. Additionally, their environment forces them to add more crappy code to the existing crappy code, making the situation worse for the future and multiplying their frustration.
• Operations are unhappy because they are confronted with a (hopefully) feature-complete, yet not production-ready solution. The solution handed over typically is brittle, buggy and lacks most features required for reliable operations. And whenever they demand that some minimal operations needs are met, they are accused to be the “department of ‘No’”.
• Management is unhappy because they cannot shine with such projects. Especially IT leaders regularly need to explain themselves to non-IT leaders.
• And so on …

And all that because people do not understand software and are driven by all those misconceptions in their head.

The road ahead

How can we do better?

Well, for starters, we could try to identify all those practices, routines and habits that work against the properties of software and replace them with new practices, routines and habits that support the unique benefits of software while taking its constraints into account.

Unfortunately, in many companies this list would comprise almost everything they are doing related to software. E.g., my whole “Simplify!” blog series mainly discussed shortcomings we can observe in many IT organizations which are direct or indirect results of not understanding software.

But in the end, I think this is the only sensible choice if we want to end up with a “more productive” IT:

• We need to understand software and leave the misconceptions behind us.
• We need to think from first principles how to leverage the unique strengths of software without violating its constraints.
• This will most likely result in a target organization, processes and practices that are very different from what we have today.
• And then we need to change …

Yes, change is hard. I wrote about it before. There we be resistance. There will be sentences like: “We know that our current processes are not perfect. But at least they work.” – and all those other things that people do to evade change from playing dead to aggressively threatening the change advocates.

But what is the alternative?

Trying to squeeze a few percent more efficiency out of the software development process by introducing AI solutions while the process itself is deeply flawed and deprives us from everything that makes software a unique material that “ate the world”? ³

Really?

I leave it to Russell Ackoff to counter that idea:

“The more efficient you are at doing the wrong thing, the wronger you become. It is much better to do the right thing wronger than the wrong thing righter. If you do the right thing wrong and correct it, you get better.” ⁴

Things do not become better if you do the wrong thing more efficiently. They only become better if you correct them, i.e., start to make them right. So, we are back once more to the point that the challenges is not to become more efficient but to become more effective.

I will not add a detailed list how to do things. I already added a few counter-examples in this blog series you can use as a starting point if you like. A full list would require a whole blog series on its own – like the aforementioned “Simplify!” blog series which adds a lot more ideas. And most likely, there is not a single list how to do things right. It will also depend to a certain degree on external factors like the industry you are in, how big you are, and alike.

However, I think the list is not the most important thing. I think, the most important thing is to understand the widespread misconceptions regarding software and part with them. Only if we really understand the uniqueness of software with all its strengths but also its caveats, we will be able to leverage its full potential. Hence, let us start there …