step 18 / 32 · 6 min
When the answer depends on who you ask
One database on one machine has one version of the truth. Add a replica, a cache or a second service and that stops being true, and most of the difficulty in distributed systems follows.
Strong and eventual
Strong consistency: every read returns the most recent write. Simple to reason about, and it requires coordination, which costs latency and availability.
Eventual consistency: reads may be stale, and converge given time. Cheap and fast, and the application must tolerate seeing old data.
Neither is better. A view counter can be eventually consistent. An account balance cannot.
Read your own writes
The specific case that produces bug reports:
user updates their profile -> written to the primary
page reloads -> read from a replica
the replica is 200ms behind -> they see the old valueThe user did everything right and the system appears to have lost their work.
Fixes, in order of preference: read from the primary for a short window after a write by that user, or route that user's reads to the primary, or wait for the replica to catch up before responding.
This guarantee has a name, read your own writes, and it is usually the only one users actually notice the absence of.
CAP, stated narrowly
During a network partition, a system must choose between refusing requests and serving possibly stale data.
That is all it says. It is not a claim about normal operation, and it does not say you must abandon consistency in general. It is quoted far more broadly than it holds.
Transactions across services
A monolith wraps two changes in one transaction. Two services cannot.
Two phase commit coordinates them and blocks if the coordinator fails, which is why it is rarely used at scale.
Saga: a sequence of local transactions, each with a compensating action if a later step fails.
reserve stock -> charge card -> confirm order
fails
release stock <- compensateThe system is briefly inconsistent, and compensation is not the same as a rollback: a charge that was refunded still happened and is visible on a statement. Designing what the user sees during that window is part of the work.
Idempotency again
Retries and at least once delivery mean an operation may run twice. A key generated once by the caller and recognised by the server is what makes that safe, and it is the single most valuable habit in distributed work.