Eventual Consistency: What It Means in Practice
A distributed system cannot be instantly consistent everywhere. Learn the models, the read-your-writes problem, and the fixes.
By Velloreth
What it means
A write is applied to one replica and propagated to the others.
The replicas converge, but they are not instantly equal.
The window between the write and the convergence is the inconsistency window.
The window is usually milliseconds to seconds, and it is visible when the user reads immediately after a write.
The models
| Model | Guarantee |
|---|---|
| Strong | a read always sees the latest write |
| Eventual | the replicas converge |
| Read-your-writes | a user sees their own write |
| Monotonic read | a value never goes backwards |
| Causal | a cause is seen before its effect |
| Bounded staleness | the lag is under a limit |
Strong consistency costs latency, because a write must be acknowledged by every replica or a quorum.
Eventual consistency is faster, and the application must handle the window.
Read-your-writes
The problem users notice most.
1. The user updates their name
2. The write goes to the primary
3. The user's next read goes to a replica
4. The replica has not caught up
5. The user sees the old name
It looks like a bug, and it is the most common complaint about a read replica.
The fixes:
| Fix | How |
|---|---|
| Read from the primary | after a write, for a short period |
| Sticky routing | the user reads from the same replica |
| A version check | wait until the replica has the version |
| An optimistic UI | show the change immediately |
The sticky routing is the cheapest, and it requires the user to be identified.
Monotonic reads
A value must not go backwards.
1. A read from replica A returns version 5
2. A read from replica B returns version 3
3. The user sees an older value
The fix: route a user to the same replica, or track the version and reject an older one.
A refresh that shows an older value is confusing, and it is a common bug in a load-balanced read.
The quorum
A majority of the replicas.
A write is acknowledged when a quorum has it.
A read consults a quorum.
The read and the write overlap, so the read sees the latest value.
N = 3 replicas
W = 2 (a write quorum)
R = 2 (a read quorum)
W + R > N -> the read and the write overlap
A larger quorum is more consistent and slower.
The tunable consistency of Cassandra and DynamoDB is this trade-off exposed as a parameter.
The patterns
An optimistic UI. The change is shown immediately and reconciled when the server confirms.
A pending state. A value that is not yet confirmed is shown as pending.
A read from the primary after a write. A short window, keyed by the user.
A version in the response. The client tracks the version and rejects an older one.
A causal token. The write returns a token, and the read passes it, so the replica waits until it has that version.
A bounded staleness. The read waits if the replica is more than a limit behind.
When it is acceptable
Acceptable:
- A like count
- A view count
- A feed
- A search index
- A recommendation
- An analytics dashboard
Not acceptable:
- A balance
- An inventory count
- A unique constraint
- A permission check
- A payment status
The test: does a stale read cause a wrong action? If yes, the read must be consistent.
The UI
Design for the window, or remove it.
The techniques:
- Show the change immediately, and reconcile
- Show a pending indicator
- Read the user’s own data from the primary
- Poll until the value is confirmed
- Show a relative time, so a small lag is invisible
A user who sees their own change immediately does not notice the window.
Common mistakes
A read replica for a read-your-writes case. The user sees the old value.
No sticky routing. A value goes backwards.
A quorum that does not overlap. The read misses the write.
A stale read for a permission check. A security bug.
A stale read for an inventory count. An oversell.
No pending state. The UI looks broken.
A cache with no invalidation. A longer window than the replica.
A test that assumes strong consistency. A flaky test.
No monitoring of the lag. The window grows unnoticed.
A quotable summary
Eventual consistency means the replicas converge rather than being instantly equal, and the window between the write and the convergence is usually milliseconds to seconds. Read-your-writes is the property users notice most, and the fixes are to read from the primary after a write, to route the user to the same replica, or to update the UI optimistically. A quorum read trades latency for freshness, and a stale read is acceptable for a count and not for a balance.
Related reading
For the related topics, see cap-theorem-explained and database-sharding-basics. For the broader context, caching-strategies-guide and background-jobs-guide.