A note about eventual consistency - Part 1

Recently I saw a skeet on Bluesky:

“Eventually consistent mainly means it will be regularly non consistent at all.”

I deliberately leave out the link to the post because I do not have the faintest idea regarding the context and motivation that made the author write that skeet and I do not want to make any assumptions about it. The author could have meant a lot of things.

However, just reading this skeet without any additional explaining context reminded me of many misunderstandings regarding eventual consistency I encountered over the years and it motivated me to write this post – or actually two posts as it became too long for a single post – where I will try to shed some light on a (still) massively misunderstood topic.

Lost in translation

First of all, I would like to clarify a huge misunderstanding I encountered time and again in Germany, where I live. I also encountered it in some other non english-speaking countries. The misunderstanding stems from a different meaning of two similar sounding words in different languages.

Here it is “eventually”. The word looks and sounds a lot like the German word “eventuell”. Actually, when looking at their spelling, both words obviously stem from the same origin. However, they mean something very different in English and in German. While the English word “eventually” translates to the German word “schlussendlich”, the German word “eventuell” translates to the English word “potentially”.

This means that most Germans who hear “eventually consistent” understand “potentially consistent”. I even heard people using the German phrase “eventuelle Konsistenz”, i.e. “potential consistency” when talking about eventual consistency. People who make this implicit mistranslation think that consistency in an eventually consistent database is a matter of chance, sort of: If the write is successful, it is fine, if not, it is fine, too.

However, reality is that “eventually consistent” is a clear consistency guarantee. It just do not guarantee atomic and isolated updates as ACID transactions do. Thus, not “eventuell konsistent” but “eventually consistent”.

Lost in translation!

Eventual consistency is not that different

Having clarified this translation pitfall I often encountered, I think we should discuss first what “eventually consistent” actually means before moving on to other misunderstandings.

Eventual consistency is a consistency guarantee. The semi-formal way of describing the guarantee is that if no further updates should occur, all updates across all nodes involved will eventually converge to a consistent state.

At first sight, this does not look too promising and may lead to the interpretation that most of the time everything is out of sync, i.e., inconsistent. However, it is important to keep in mind that such formal definitions are typically phrased very cautiously because they also need to be valid for edge cases, e.g., if a longer-lasting network failure including split-brain situations occurs or some other quite rare exceptional cases.

Under normal conditions, eventual consistency mostly feels like strong consistency: You see an old state of the data, you apply an update, you see the new state – no big delays, no long-lasting interim inconsistent states, nothing. Just like with ACID transactions.

Also minor hiccups in the network or the other parts of the infrastructure are not a problem: With ACID transactions, it takes a bit longer until the result of the update becomes visible, with BASE transactions (simply put, this is how eventually consistent updates are usually called), there may be a short period where we might see a temporary inconsistent state which is gone shorty after the hiccup is over. Normally, we also do not experience differences if the hiccup is short.

The differences only show in the relatively rare cases if some bigger issue hits the database system, like, e.g., a longer-lasting network partitioning, a massively overloaded or crashed server, or alike. In such situations, strongly consistent database favor consistency over availability while eventually consistent databases favor availability over consistency which leads to a different behavior, including different tradeoffs.

However, this different behavior persists only while the adverse situation exists. After being back to normal conditions, both types of databases will converge back towards the same state and with it to the same perceivable behavior. ¹

Progress and safety in distributed systems

To understand this different kind of behavior under adverse conditions and its tradeoffs better, we need to look at progress and safety ², the two eternally competing properties of distributed algorithms that permeate the literature and research of distributed systems:

• Progress means we want to ensure that our system is not stuck indefinitely, that it will respond and/or update its state within a defined maximum time even if that means that safety, i.e., perceived consistency or response accuracy may suffer. Put simply: Better an approximate answer in time than a perfect answer that never arrives.
• Safety means we want to ensure that our responses and/or state updates are always exact (consistent and accurate) even if that should mean it may take an indefinitely long period of time to get there (e.g., if a required node crashes for good). Put simply: Better no answer at all than a non-exact answer.

Those two properties are more general properties than availability and consistency but in general availability is related to progress and consistency to safety.

Both system properties are desirable, timely responses and response accuracy. However, you can only ensure both of them as long as everything works fine: No failing nodes, no network partitions, no big latency (for whatever reasons), etcetera. Under such conditions, both properties can be fully satisfied.

But as soon as we encounter a (hopefully) temporary adverse situation where those nice conditions are not given anymore, we need to decide if we value progress over safety or vice versa. This decision defines which route we take and how we need to design our system. In our case it defines which kind of consistency model we need to use – including accepting its tradeoffs.

When eventual consistency is different

Strongly consistent transactions lean towards safety: They rather stall indefinitely or abort than allowing an update that could exhibit even a temporarily inconsistent state. Then upside is that temporarily inconsistent states are not exhibited. The downside is an availability hit if conditions are outside the “expected”. Typically, you cannot write to the database at all until the adverse situation is resolved. Depending on your business requirements, this can be intolerable.

Then you rather choose the route of eventually consistent transactions: To maximize availability, the database allows partial updates. It may update only a subset of the involved nodes (especially if it uses replication what many eventually consistent databases do) and cannot reach all nodes, e.g., due to a network partition, a node failure or a bad case of latency. The non-reachable nodes will be updated later after they become reachable again.

Depending on the quorum settings, i.e., how many acknowledged writes and reads are required per update or read across the replicas, you can still expect to receive either accurate data or maybe no data – at least, if they are set up correctly. ³

Eventually consistent databases also break up the atomicity and isolation properties of ACID transactions. With ACID transactions you can bundle a series of logically connected updates that together create a new consistent state within a transaction. The ACID properties then make sure that you either see the consistent state before the transaction begins or the consistent state after the transaction completes from the outside. You do not see any interim inconsistent states from the outside while the transaction is still ongoing.

Eventually consistent databases use so-called BASE transactions that do not support atomicity and isolation across multiple updates. The reason again is that eventually consistent databases favor progress over safety. Atomicity and isolation across multiple updates are very nice properties but they come at a price. The price is a progress hit in favor of safety. If an adverse situation occurs, ACID transactions tend to stall or abort. Both responses mean “no progress”.

As eventually consistent database value progress over safety, they do not support atomicity and isolation across multiple logically connected updates. Instead, each of the logically connected updates is executed on its own. This usually limits atomicity and isolation to a per node and update level, sometimes even to a per entity and node and update level.

The latter one means that we can expect that a single entity (i.e., a table row or a document) on a single replica is updated in an all-or-nothing fashion. But if we update more entities on a single node, the updates cannot be expected to be executed atomically and isolated. The same is true if we update a single entity on multiple nodes.

The exact guarantees an eventually consistent database provides depend on the way the database implements its update logic internally. Some eventually consistent databases provide higher-level guarantees than just per-entity-and-node-and-update. However, in general, we cannot expect higher-level guarantees and even if they are higher, they will not be ACID-level guarantees.

This still does not mean that we will necessarily see inconsistent interim states. Usually, we will not as updates to a single node normally complete very fast as most eventually consistent databases support bulk updates.

Sending the logically connected updates to the database nodes at once without needing to employ a remote call for each update reduces the probability of seeing an interim state from another client significantly – simply because the updates are executed much faster and the probability of a partial update failure due to a network issue is much lower. This way, we usually either see the consistent “before” state or the consistent “after” state if we query the database.

Additionally, “entities” (i.e., atomically updated units, not to be confused with entities in the relational paradigm) are usually designed at a much more coarse-grained level than in relational ACID databases which also reduces the risk of seeing an inconsistent interim state of multiple logically connected updates.

Nevertheless, all this does not eliminate the risk of seeing an interim inconsistent state of an ongoing series of multiple logically connected updates. If we are unlucky, we may see one. The database does not protect us from seeing them.

It can also happen that when an adverse situation started, the database node we are about to query was in the middle of a logical transaction that transfers the database from one consistent state to the next. Some of those logically connected updates have been applied already while some are still outstanding because they cannot be sent to the node due to a network partitioning or alike. Then, we may see an inconsistent interim state if querying this node as long as the adverse situation persists.

This is the price we need to pay if we value progress over safety (or in our more specific context: availability over consistency): We may see inconsistent interim states, i.e., a mix of old and new state in a set of logically connected entities. We also may see temporal different states if we send the same query to multiple nodes.

However, while this can happen, it happens far less often than most people think if they hear “eventually consistent”.

Consistency is not an either-or

Note that strong (ACID) and eventual (BASE) consistency as well as safety and progress are not an either-or. They are rather a spectrum, we can fine-tune based on our specific needs. There may be situations where we need strict safety. In other places, some exceptions are okay to improve progress. And sometimes, progress is all that matters.

We understand this best if we look at the different consistency guarantees we find in the literature as well as in the wild. It is not just ACID or BASE. Instead, there are dozens of different guarantees that span a wide spectrum between perfect safety (giving up any progress guarantees) and no safety at all (to maximize progress), all with their specific guarantees and tradeoffs (see, e.g., “Highly Available Transactions: Virtues and Limitations (Extended Version)” by Peter Bailis et al. for a compilation and discussion of multiple consistency guarantees across the whole spectrum).

Summing up

While there are a lot more details and nuances that would be worth discussing when it comes to eventual consistency, I will leave it here for the moment. After dismantling the lost-in-translation trap, we see in some countries like, e.g., Germany where I live, we discussed what eventual consistency actually means and why we sometimes decide to favor eventual consistency over strong consistency.

Most importantly, we have seen that – if implemented correctly – most of the time we will not perceive any differences between eventual and strong consistency, that the differences only become apparent if the system encounters adverse conditions.

Still, eventual consistency seems to have a negative connotation for many people. Why?

This is the question, we will dive into in the second (and final) post of this little series (link will follow). Stay tuned … ;)