A note about eventual consistency - Part 2

In the previous post we discussed what eventual consistency actually means and why we sometimes need to favor eventual consistency over strong consistency. We also saw that most of the time we will not perceive any differences between eventual and strong consistency if set up properly. The differences only become apparent if the system encounters adverse conditions like, e.g., a network partition, loss of a node, or alike.

Still, eventual consistency seems to have a negative connotation for many people. In this second (and final) post of this little series, we will examine where this negative connotation may stem from.

Let’s get started …

Misleading interpretations and promises

A big part of the negative perception regarding eventual consistency probably stems from the fact that many “eventually consistent” databases are not eventually consistent but merely best-effort consistent, or accidentally consistent as I tend to phrase it.

Eventual consistency does not come for free. It usually requires well-configured and well-implemented quorum-based reads and writes as well as a good reconciliation mechanism to reliably reconciliate nodes after a network partition or other kinds of failures that lead to out-of-sync nodes.

Some of the most popular NoSQL databases like Cassandra and MongoDB did not provide these settings out of the box. E.g., Cassandra, being optimized for maximum availability, does not provide any consistency guarantees across its nodes out of the box. It is possible to crank up the settings to ensure eventual consistency. However, out of the box, it does not guarantee eventual consistency.

MongoDB before version 5 also favored update speed over consistency. Especially in the beginning, MongoDB boasted to be blazingly fast regarding updates. And they were – at the expense of consistency guarantees. They also did not provide any consistency guarantees using their default settings. Again, it was possible to crank up the settings to ensure eventual consistency but out of the box eventual consistency was not guaranteed.

With version 5, the default became a read and write concern (that is how they name it) of “majority” which could be sufficient to ensure eventual consistency ¹. Nevertheless, you can still shut off eventual consistency easily by setting different concerns. But at least, this is your own decision now and is not due to database default settings (which are often left untouched).

If you used (or use) a database not set up for eventual consistency but best-effort consistency, chances are very high that you will quite regularly experience data inconsistencies. A minor network hiccup or a temporarily latent node often is enough for the replicas to run out of sync or losing updates altogether.

Use cases and consistency decisions

To be clear: This is not a flaw of the aforementioned databases. They were (and are) quite clear about their design goals, and high consistency was (or is) not a major goal.

Cassandra basically was a column-oriented reimplementation of the Amazon Dynamo paper. The primary design goal of Amazon’s Dynamo and thus also Cassandra was high availability: As long as a single node of the database was still up, the database should accept write requests.

This design goal and the resulting consistency properties become perfectly clear, if you understand that Amazon originally built Dynamo as storage solution for their shopping cart:

Before Dynamo, they used a different solution – probably one of the traditional RDBMS. Unfortunately, under very high load (which Amazon experienced more and more often), the old solution ran into availability problems: It did not accept all “Put into shopping cart” requests.

This was a significant business problem. Amazon knew that every dollar put into the shopping cart meant X cents revenue ². If the shopping cart storage did not accept writes, it meant they lost real money, plus indirectly some more money due to dissatisfied customers and the negative reputation derived from it. This problem was big enough that they decided to design and build a new database from scratch with a single major design goal: Accept writes as long as any node of the database is still up and running. Everything else was secondary.

This also meant that Consistency was not that important. The business level reasoning for this decision was like that:

Typically, users only add items to their shopping cart before they check them out. Only quite rarely customers remove items from the shopping cart after adding them. They also only rarely modify items in the shopping cart (i.e., change their quantities) after adding them.

If customers only added items to their shopping carts, reconciliation after a partial failure was easy ³. If customers removed items during a partial failure, it could happen that removed items reappeared after reconciliation. The same was true if customers changed quantities of items in the shopping cart during a partial failure. It was possible that the old quantity showed up after reconciliation. In short: The default use case was eventually consistent, all other were not.

Nevertheless, this was not considered a major issue: Such potential inconsistencies occurred only quite rarely, and if they occurred usually the customers noticed it during checkout, i.e., if deleted items reappeared in their shopping cart or quantities were wrong. Then the customers typically simply fixed the issue by removing the extra item or correcting the quantity.

And if the customers did not notice – well, then Amazon apologized, took back the surplus articles in a hassle-free way and added a voucher as consolation. From an economic perspective this was a lot better than losing lots of revenue because the storage did not accept writes under high load.

In short: Based on the use case, Amazon’s Dynamo was created for, it should be clear why they favored availability over consistency – and Cassandra, being a column-oriented reimplementation of Dynamo, did the same.

MongoDB had a similar origin story. IIRC, it was originally built as a solution to store log entries that came from the fire hose and make them searchable for log analytics (this was in a time before all the nice log aggregation and analytics solutions existed we are used to today).

Most storage solutions that offered an acceptable query functionality were not prepared for drinking from a fire hose. They simply did not accept writes quick enough. And most storage solutions that were fast enough to drink from the fire hose did not offer acceptable query functionality. E.g., writing to a file system was quick enough but ad hoc queries on huge files – well – sucked.

As a consequence, they designed a novel database that combined very fast write capabilities with good ad hoc query functionality. They also implemented replication capabilities across multiple nodes to offer some degree of fault-tolerance, plus query speedup options by querying read replicas, IIRC.

However, to process the incoming writes fast enough, they also had to compromise consistency. They acknowledged a write after the master node had received it and had written it to a memory mapped file. Not yet written to a disk, only to memory. Not yet replicated across a cluster majority, only accepted by a single node.

This made MongoDB blazingly fast and it was able to handle very high write loads – at the expense that it might lose some updates if things did not work as expected. But this was acceptable with the use case it was originally designed for. Losing a few log entries due to occasional network hiccups or latent nodes was a lot better than losing many log entries because the storage solution was not able to process the amount of incoming writes – or even worse, blocking the sending applications because their sent log entries queued up.

Again: Based on the use case, MongoDB was created for, it should be clear why they favored availability over consistency – at least in their early years.

However, both databases – Cassandra and MongoDB – became highly popular. The companies behind those databases received a lot of VC backing, including the revenue expectations attached to them. To satisfy the revenue expectations, the companies started to aggressively advertise their databases as general purpose databases, as sort of a drop-in replacement for the “boring” relational databases.

As a consequence, both databases were used for a much broader set of use cases, often including mission-critical data. But their default consistency settings were still based on their original use cases and the resulting design decisions. Again, the design decisions were perfectly valid based on the use cases they based their implementation upon.

But if those databases were used as a “drop-in” replacement for a relational database with its ACID transactions as many people did (and sometimes still do), those people were (and still are) in for some very unpleasant surprises regarding consistency.

Do not believe. Verify!

Such experiences were not limited to Cassandra and MongoDB. I only used them as examples because they were the two most popular NoSQL databases and thus most unpleasant surprises regarding “eventual consistency” probably stem from using one of them without carefully adjusting their consistency settings.

A lot of other databases also did (and do) not offer eventual consistency out of the box. Hence, unpleasant surprises regarding consistency were (and are) also the norm if using these databases as if they implemented ACID transactions.

Therefore:

Always read the documentation (not the marketing materials!) very carefully to understand which consistency guarantees a database gives you out of the box and how and up to which level you can tune the consistency guarantees.

And maybe even more importantly:

Never choose a database (or any other product) based on their marketing materials.

Corollary: Everything on the product web site (including blog) except the actual product documentation must be considered marketing material.

To sum up: Not all “eventually consistent” databases are eventually consistent out of the box. Depending on their setup, often they are just “accidentally consistent”, i.e, implement some kind of best-effort consistency. Based on my experiences, this is one big reason why eventual consistency has such a bad reputation. ⁴

The intricacies of eventual consistency …

The other big part of the negative perception regarding eventual consistency most likely stems from the fact that from a developer’s perspective eventual consistency is quite a bit harder to handle than ACID-based strong consistency. Reasoning about an ACID transaction is quite easy:

• There is a consistent state before the beginning of the transaction.
• All updates that are needed to ensure a new consistent state are either executed completely (commit) or not at all (abort).
• The interim, potentially inconsistent states that arise while the updates are applied, are not visible outside the transaction.
• After the transaction completes, we have a new consistent state.

This means, as a developer you do not need to ponder potentially inconsistent interim states and how to deal with them if you use ACID transactions. In your mental model, you just move from one consistent state to the next. Everything is consistent, all the time. Straightforward and easy to understand (in practice, things are not that easy but more about that later).

With eventually consistency on the other hand, you need to expect temporary inconsistencies. Even if they do not occur often, they can occur. This means, we always need to take them into account while pondering our designs and implementations. And this can be a real mind twister. Imagine two pieces of data that are related to each other but are not retrieved by the same query and/or are not modified by the same update. Whenever we process them together in our application they might be out of sync, i.e., they might not fit together at that point in time. This has a lot of implications for our application design:

• We need to figure out how to detect if data is currently out of sync.
• We need to figure out from a business level perspective how to proceed in such a situation because usually simply signaling an “internal error” is not acceptable.
• We need to implement the respective corrective measures or at least mitigation measures.
• In the worst case, i.e., if we cannot rely on the database to ensure eventual consistency (see the previous sections), we even need to implement the required reconciliation measures.

Yikes! This is a lot harder to get right than simply moving from one consistent state to another and leaving it to the database not to expose any inconsistent interim state. This can be exhausting. Especially if you think about it for a while, you will realize that a lot of undesirable combinations of temporary inconsistencies are possible which all need to be detected and handled.

We can reduce the number of potential issues and combinations we need to ponder significantly by employing different database schema design techniques (which are outside the scope of this post). We can also redesign and rearrange our application logic to reduce the number of potential issues we need to handle (which is also outside the scope of this post).

Nevertheless, the problem will persist and needs to be addressed because the only alternative is to make incorrect decisions based on temporary inconsistent data – if we are not “lucky” enough that the program will crash because of the inconsistency before we are able to make the incorrect decision.

The incorrect decision will then spread beyond the boundaries of our application and eventually lead to persistent inconsistencies, i.e., persistently messed up databases. Hence, not an actual alternative.

… and the consequences of ignoring them

However, many people were not aware of this when they went for eventual consistency for the first time. Actually, for a quite a while using an eventually (or “accidentally”, see above) consistent database instead of a “boring” relational database that offered ACID transactions was all the rage. You were “so uncool” if you were still stuck with a RDBMS.

Very few of those “cool” people took into account that switching from strong consistency to eventual consistency meant new, previously unneeded and quite complicated stuff they needed to take care of. Typically, they did not know anything about consistency. For them it was something that was simply a “given”. They never had to ponder consistency in their whole careers because the “boring” ACID-based relational databases took care of it.

And hence, they designed and implemented their systems as if the underlying database were still offering strong consistency. As a consequence, they were in for some very unpleasant surprises regarding consistency.

The case for strong consistence

Does all that mean we should dismiss eventual consistency? Well, no. There are use cases that require eventual consistency. And then, we should go for it. We just need to make sure to get it right – which often requires a lot of hard thinking.

However, we should not go for it recklessly, just because that eventual consistent database feels cooler, just because we consider ourselves “big data” while we are not, or alike. As discussed before, eventual consistency comes at a price, and the biggest part of the price is the much harder reasoning model. Based on my experience, the most important reason for using ACID transactions whenever the use case allows it is exactly the much simpler reasoning model for software engineers. Eventual consistency is so much harder to reason about. ⁵

Therefore, I always recommend with a lot of emphasis:

Never switch from strong to eventual consistency without a good reason.

Addendum: Fashion, hubris and CV-driven development are very bad reasons.

To sum up: Eventual consistency is a much more demanding mental model for developers. Based on my experiences, this is the other big reason why many people had bad experiences regarding eventual consistency. They grew up only knowing ACID databases and then were suddenly faced with the intricacies of eventual consistency – including the unpleasant surprises of ignoring the intricacies.

Side note: Also ACID transactions usually are not what most people think they are. Most people have serializability in mind, the highest isolation level, ACID transactions define. However, in practice most database run at lower isolation levels for performance reasons which allow for several kinds of data anomalies and inconsistencies. In other words: You also need to keep possible anomalies and temporary inconsistencies in mind if you work with ACID-based transactions, even if they are less likely than temporary inconsistencies in the context of eventual consistency.

I wrote a whole 4-part blog series about the fallacies regarding ACID transactions. Hence, I do not repeat them here in further detail.

Summing up

Eventual consistency still is a highly misunderstood topic. It starts with the fact that the word “eventual” and similar looking and sounding words in other languages can have quite different meanings (like, e.g., the German word “eventuell”), leading to a lot of confusion.

There is also the misunderstanding that eventually consistent means that databases are inconsistent most of the time because the formal definition of eventual consistency is quite cautious. Truth is that under normal conditions, i.e., if the database cluster is up, running and all nodes are connected to each other, the database basically is (almost) as consistent as a ACID-based database that provides strong consistency.

The differences only become apparent if a problem like a node or network failure hits: While strongly consistent systems then usually refuse to accept further updates to ensure consistency, eventually consistent systems still accept updates (within certain bounds) to improve availability. There are good reasons for both options and depending on the given context, you will prefer the one or the other and need to accept its tradeoffs.

However, eventual consistency has a bad reputation among quite a lot of people. Based on my experience, the most important reasons for this are

• people who unknowingly used databases that were not set up for eventual consistency but “accidental”, i.e., best-effort consistency
• the neglected or underestimated additional complexity regarding the mental reasoning model that eventual consistency requires

both typically resulting in eventual persistent inconsistencies.

While the database issue can be fixed using the right database settings, the added reasoning complexity of eventual consistency cannot be avoided. Going for eventual consistency means accepting the significantly higher mental load (or accepting to make incorrect decisions once in a while with all its implications). Therefore, my recommendation regarding eventual consistency is:

Only use eventual consistency if you have a good reason for it.

This does not mean that going for eventual consistency is a bad idea per se. There are use cases where going for eventual consistency is the only sensible choice. But it results in a significantly higher mental load – and it only works if we set up our database correctly. Otherwise, we only get “accidental consistency”. Therefore, do not use eventual consistency just because “you can” (or think it is “cooler” than the alternative which provides strong consistency).

I hope, these two posts helped to shed some light on the still massively misunderstood topic of eventual consistency and to get rid of some of the many widespread misunderstandings regarding the topic. Maybe the post even supports you a bit when it comes to making a decision between strong and eventual consistency. I would be glad if it would …