Quality - The unknown entity - Part 1

Quality is the subject of countless discussions, e.g., with unsatisfied clients. ¹

I had the situation more than once that a client was not satisfied, that they felt their quality expectations for the software were not met. When I then went to the development team and asked them about quality, it often turned out that they did not understand the reproach. In their perception, quality was good.

How could that be? Who was right and who was wrong? Well, both parties were right from their point of view. But content-wise they talked about totally unrelated topics.

While the clients, e.g., talked about response times, user experience and issues becoming apparent in pen testing, the developers focused on test coverage and code quality.

So, quality seems to be an at least somewhat ambiguous topic. In this little post series, I will discuss a few observations, I find important regarding “quality” and discussions about quality.

• This first post discusses the subjectivity and multi-dimensionality of quality.
• The second post discusses that we need to make quality measurable to create a shared understanding. It also discusses that many people do not understand how their demands affect quality, and that quality attributes can be conflicting.
• The third post discusses that quality cannot be judged in isolation but that it always lives in a context that needs to be considered.
• The fourth and last post discusses that quality needs to be balanced on multiple levels. It also discusses that it temporarily can be okay to compromise quality to reach a more important goal, and that just watching what you do not want (e.g., bugs) does not necessarily lead to quality.

Let us start here with the first observation, that quality is subjective.

Quality is in the eye of the beholder

The probably most important observation regarding quality, even more important than the question, what quality is, is:

Quality is in the eye of the beholder

It is crucial to understand that quality is in the eye of the beholder. As the Wikipedia article regarding quality in a business context states:

“Quality is a perceptual, conditional, and somewhat subjective attribute and may be understood differently by different people.”

As a consequence, a single “correct” notion of quality that is true in every context does not exist ². The brief example at the beginning already pointed it out: The client had a very different understanding what quality is than the development team had.

The best we can achieve is a shared quality understanding by the people involved in a given context. Still, it is important to understand that this shared quality understanding is context-specific, i.e., you cannot simply transfer this understanding to another context.

The only way to create a dependable shared understanding of quality is to make it explicit. You need to discuss with the stakeholder groups involved in a project what properties they expect from the software – under development and at runtime.

It is important to be aware that your “client” usually is not a single, homogeneous stakeholder group, even if you were commissioned by a single party. The “client” typically consists of a dozen or more parties in most projects, e.g., business department(s), management, development team, quality assurance, operations, finance, marketing, procurement, etc. – and of course the different user groups of the resulting system, way too often neglected in the process of creating the solution.

Each group has different needs and demands regarding the software solution and thus different expectations regarding quality. It is important to understand the quality expectations of the different parties, to make them explicit and to create a shared understanding actively. Otherwise, you most likely will suffer from contradicting expectations that – in the worst case – will jeopardize the entire project.

I will come back to making quality explicit later.

Quality has many dimensions

But first let us discuss a different question:

What is quality?

After having understood that a single view on quality does not exist, it is important to understand what quality actually is to make the topic discussible.

Quality is not a single value that is “good” or “bad”. Instead, quality has many dimensions. Each dimension also is not just “good” or “bad” or a degree of fulfillment between 0% and 100%. Instead, each dimension usually is described with a set of differentiated requirements or scenarios (we will come back to these scenarios in a later post).

But what are the dimensions of quality?

(Not only) in software development, these dimensions are described by so-called “quality attributes”.

The figure lists some common quality attributes for software-based solutions:

• Accessibility
• Availability
• Effectiveness
• Efficiency
• Evolvability
• Fault Tolerance
• Functionality
• Installability
• Interoperability
• Learnability
• Maintainability
• Observability
• Operability
• Portability
• Recoverability
• Reliability
• Resource utilization
• Scalability
• Security
• Time behavior
• Understandability
• Usability
• Usefulness
• …

This is a subset picked from several software quality standards like ISO/IEC 9126 and its successor ISO/IEC 25010 as well as other sources like the quality attribute description of the SEI or the list of system quality attributes that partially also applies to software.

Note that the list is not complete. Based on my experience a single standard that perfectly describes all quality attributes relevant for software does not exist (even if all the existing standards claim to do so). At least in my projects I always had to mix and match attributes from several sources to capture the quality needs best.

Trying to capture all requirements for a given standard dogmatically typically annoyed the clients, as many of them were not relevant for them in their context. Additionally, some really relevant attributes were always missing, and I had to add them to the list from other sources. Typically, I pick the most relevant attributes for a given context which usually ends up in a list of around a dozen attributes. Depending on the client and its experience with quality attributes, I either pick the attributes together with the client or I do it on my own.

But even if we are able to limit the relevant quality attributes to just a dozen in a project context, we can observe that “quality” still is a multi-dimensional beast. As written before, we cannot simplify quality to a binary evaluation like “good” or “bad” or a degree of fulfillment between 0% and 100%. There are always multiple dimensions we need to look at.

That is the most common source of misunderstandings and conflict, like the one sketched at the beginning of this post. The development team did not understand why the client was not satisfied with the quality of the solution because they put so much effort into unit testing and general code quality (by, e.g., applying clean code principles).

Yet, as we can see in the figure above, unit test coverage and code quality only affect a small part of the overall quality attributes, namely understandability, maintainability, functionality and evolvability. Depending on the actual implementation of unit tests and code quality, they even affect these attributes only partially.

Unfortunately, the affected quality attributes were not the attributes that were most important to the client. Therefore the client perceived the software solution quality as “bad” while the development team perceived it as “good”.

As written before, both parties were right (and wrong) at the same time. Still, there is an unwritten law which says:

The person who pays the bill defines what is “right”.

In software development, this usually is the client. Thus, I personally think that it was a bad idea of the software development team to focus solely on the quality attributes most relevant for them while neglecting the needs of their client.

To be clear: Topics like caring about code quality and unit test coverage definitely have a significant value. Neglecting them is a bad idea, either. Especially, as software needs to be changed regularly to preserve its value, it is important to make sure that these changes remain as easy as possible.

But limiting quality concerns to these topics is by far not sufficient.

The quality needs of the client need to be taken into account and need to be made explicit. On the other hand, clients not being proficient in software development also need to understand why the quality needs of the development team are relevant.

That is why we need the discussion and why quality needs to be made explicit. We will discuss this topic in the next post.

Summing up

In this post we have discussed that quality is subjective and that quality has many dimensions. Thus, it is important to make quality explicit, to understand what the different people mean exactly if they want “high quality”. Otherwise, trouble and frustration are guaranteed.

In the next post, we discuss that we need to make quality measurable to create the required shared understanding. We also discuss that many people do not understand how their demands affect quality (and how we need to help them) and that we need to deal with the fact that quality attributes can be conflicting.

Stay tuned … ;)

[Update] May 31, 2021: Added second footnote discussing why I tend to avoid the philosophical stance that quality is absolute and how it connects to what I have discussed in this blog post. Thx a lot to George Georgovassilis for bringing up the point as it is a relevant aspect I left out in the first version of the post.