The importance of documentation

Documentation – not only since the rise of Agile a controversially discussed topic in IT. On the one hand, we see a lot of demand for documentation by many parties. On the other hand nobody wants to create it.

The Agile documentation confusion

The rise of Agile then delivered the supposedly watertight argument for software developers who loathe to write documentation:

Working software over comprehensive documentation ¹

The typical interpretation of the second value of the Agile Manifesto was: “It is all about writing code. Documentation is irrelevant.”

Of course, the addendum was overlooked:

That is, while there is value in the items on the right, we value the items on the left more.

This states that there is value in documentation. It is not an either-or. It is about finding the right balance.

“But working software has more value”, you may contradict. Well, yes, but documentation also has value. And just delivering naked software without any documentation often reduces the value of the software. So, it is not that simple.

To get the typical misinterpretations of the Agile Manifesto out of the way, let us briefly look at the time when it was written: 2001.

The 1990 were dominated by huge project management behemoths, massive frameworks with lots of phases, tons of activities and handover points between a multitude of roles. Each handover required a specific set of documentation that needed to be created in the preceding activity.

Often you were a year or longer into a project and not a single line of software was written – but tons of documentation were created along the way.

Now remember that one of the (justified) key convictions of the people who created the Agile Manifesto was that software development is a complex task.

In (very) simple words: You need feedback from the users to get it right.

You will not get it right if you simply gather requirements upfront once, then transform them via a complicated series of completely decoupled steps into some kind of software that you finally ship. This will never result in the product the users actually wanted. ²

The problem was that all the documentation created by the big process frameworks did not help to get feedback. There was nothing that you could show to your users and ask: “Is it this what you want?”, and then gather feedback that helps you to navigate your journey to the product the users actually want.

Most of the documentation was only created to hand over information from one isolated “work station” to another of the virtual assembly line those process frameworks spanned. It was of neither intended nor suitable for collecting feedback from the users nor supporting the users or other stakeholders of the software product in any other way. Its only purpose was to feed the process.

Working software on the other hand is a great means to gather feedback from your users: You show it. Your users can use it. They can immediately experience it and give you feedback if it fits their expectations.

That is why working software has more value in Agile software development than comprehensive documentation. You need it to gather feedback, to navigate your way through a complex software development project. The creators of the Agile Manifesto wanted to emphasize that. The predominant process framework behemoths of that time did not deliver that. That is the core reason why that value has become part of the Agile Manifesto.

Good documentation

Having sorted that out, we can now look at the value of documentation.

Based on my experience, good documentation has great value – which leaves the question:

What is good documentation?

For me, good documentation has a few important properties:

• It is used to transport information across space or time - Many of the arguments against documentation revolve around the claim that direct face-to-face communication, maybe even supported by a whiteboard or alike would be a lot more effective than any written documentation.

  Admittedly, this claim is true. There is no communication form with a higher information density than face-to-face communication supported by a whiteboard or alike. The information density of written documentation is a lot worse and the likelihood of misinterpretation a lot higher.

  So, should we not better forgo documentation then?

  No.

  If you can share information face-to-face, you should definitely use this option instead of sending documentation back and forth. But what if you cannot communicate face-to-face? If direct communication with your peer is not possible, be it that your peer resides in a completely different location or if your peer does not need the communication now but someday in the future?

  Face-to-face communication is not possible in such a situation. You need to resort to a different communication means which usually means: Documentation. That is what I mean with using documentation to transport information across space or time.

  Note that this peer might even be your future self. Just because all your decisions are obvious to you at the moment, it does not mean that they will be still obvious to your future self in 6 months. A sentence or two answering the question “What the heck did I think when I did this?” that you might have in 6 months from now can be extremely helpful.

  And if your peer is a new team member who will start after you already left the team, that team member will definitely have that question and be very grateful for a short explanation.

  A last note: Many development teams only take their current intra-team communication into account when discussing documentation. Have in mind that this is only a tiny fraction of all communication needed around your software.

  There are many other stakeholder groups. Not all of them are in your room or can be brought to your room. There is a future and things that are obvious now fall into oblivion over time. Thus, there is a place for documentation – whenever you need to transport information across space or time.

• It is written with the readers in mind - Who are your readers? Are there different groups of readers? What is their background? What can you expect them to know? What do they need to know? What are their questions? Do different groups of readers have different types of questions? And so on.

  Based on my experience this property is only rarely met. Most people document from their point of view, based on their background and experience, not from a readers perspective. You need to switch perspectives to create good documentation.

  This means you might need to create different kinds of documentation for different stakeholder groups. Sometimes it might even make sense to create different documentation for a single stakeholder group facing different contexts. E.g., it can make sense to create two types of documentation for an operator: One for normal operations with all the details needed to run your application. And another one for troubleshooting, if the application is not running properly, optimized for figuring out very quickly what is going wrong and how to fix it.

  This does not mean that you need to write hundreds of pages. It is rather about concisely answering the questions the different stakeholder groups have in their particular contexts.

  Often, documentation requirements become more and more bloated over time. This is because the existing documentation neither takes the different stakeholder groups nor their respective questions into account. As the stakeholders then feel that their questions have not been answered properly, they ask for more documentation – which still does not answer their questions, and so on.

• It is concise - I guess this is self-explanatory. Documentation is not a novel – no need for lengthy introductions, off-topic deviations or fillers of any kind. Come to the point. Value the time of your readers. They will be grateful for your consideration.

  As a consequence, do not document the obvious. E.g., make sure your readers know what the upcoming documentation is about, briefly position it in the context, but do not describe the context in detail. Your readers know it.

  Keep descriptions of the “What” short. It can usually be retrieved easier in other places.

  But do not forget to document the “Why”, why you made certain decisions. Your readers need this information later if they want to change things due to new requirement. They need it to decide if they can safely change the software of if they need to leave things as they are and find a different way to implement the new requirement, if they can work around things and alike.

  Without knowing the “Why”, future decision making becomes very hard if not impossible. Unfortunately, most documentation extensively documents the “What” (because it is cheap) and neglects the “Why”.

I am sure, good documentation has more desirable properties. But documentation that keeps the few properties in mind, I described here, will be already a lot better than most documentation I usually see.

I would like to add two more remarks before wrapping up:

• Accept the information loss of documentation. Especially written documentation has a quite small information bandwidth and the chance of misunderstandings is high. The writers write in their mental context, the readers read in their context. Thus, information always gets lost or misinterpreted.

  Therefore, documentation alone usually is not enough if you really want to make sure that you bring your points across. Often you need to accompany documentation with additional measures like trainings, Q&A sessions or alike – which can be challenging if you need to transport the information across time.

• Written documentation is not the only means to transport information across space and time. While it was not easily possible in the past, today we can also transport information using audio, video and all imaginable combinations of text, audio and video up to mixed realities.

  Use your options and use them wisely. Still, the properties of good documentation are always the same, no matter which medium you use.

Summing up

Few people like to write documentation, many people are annoyed when being put off with bad (or no) documentation. Especially, in IT, documentation is a heavily debated topic since well before the agile movement.

Still, good documentation is of tremendous value for the readers and thus, we should work hard to provide good documentation where it is needed.

I tried to distill a few hints in this blog that based on my experience are essential regarding good documentation. As always I hope I was able to spark a few ideas …