Consistency, CAP, and PACELC

"Eventually consistent" is doing a lot of work in this sentence — distributed systems consistency is actually a rich spectrum, and CAP is more nuanced than the famous triangle implies.

CAP: The Original

Brewer's CAP theorem: in a distributed system, in the presence of a network Partition, you must choose between Consistency and Availability.

• C (Consistency) — every read sees the most recent write (linearisable).
• A (Availability) — every request gets a non-error response.
• P (Partition tolerance) — the system keeps working despite network failures.

P is non-negotiable in real distributed systems — networks fail. So the real choice is C-or-A under partition.

CP systems

Sacrifice availability to maintain consistency. Examples: most relational DBs in synchronous mode, Spanner, etcd, ZooKeeper, MongoDB with majority writes.

AP systems

Stay available but accept stale or conflicting data. Examples: Dynamo, Cassandra in default mode, Couchbase eventual mode.

What CAP Doesn't Say

CAP only describes the partition case. The rest of the time — when the network is healthy — both choices are possible. PACELC fills the gap.

PACELC: The Better Frame

Daniel Abadi's extension: if Partition, choose A or C; Else (no partition), choose Latency or Consistency.

PACELC captures the everyday truth: even when the network is fine, achieving strong consistency means coordinating across nodes — and coordination has latency. Many "real-time" systems are designed for low latency precisely by relaxing consistency on the happy path.

The Consistency Spectrum

"Consistent" is not binary. From strongest to weakest:

Linearisable

The strongest model. Every read sees the most recent committed write, as if there were one global timeline. Looks like a single-machine database.

• Required for: account balances, inventory at checkout, distributed locks.
• Provided by: Spanner, single-node SQL, etcd, ZooKeeper.
• Cost: cross-node coordination, higher latency, lower availability under partition.

Sequential consistency

All processes see operations in some shared total order, but not necessarily real-time order. Slightly weaker than linearisable; rarely the model anyone explicitly designs for.

Causal consistency

If write A happened-before write B (e.g. B was caused by reading A), every process sees A before B. Concurrent writes can be seen in different orders.

• Good fit for: chat messages, comment threads, collaborative editing.
• Provided by: Cosmos DB causal level, MongoDB causal sessions, some CRDT systems.

Read-your-writes

After you write something, your subsequent reads see it. Other users may not see it for a while.

• The minimum guarantee most user-facing apps need.
• Implemented by routing your reads to the leader for a few seconds, or by tracking your last write timestamp.

Monotonic reads

You never see time go backwards. If you read X=5, you don't later read X=4 from a different replica.

• Implemented by sticking a session to one replica or tracking minimum timestamps.

Eventual consistency

Given no new writes, replicas eventually converge. No bound on "eventually". This is the weakest useful guarantee — used by DNS, web caches, AP databases by default.

Choosing the Right Level

Almost no business app needs everything linearisable. The skill is matching the model to the operation:

Quorum-Based Tuning (Dynamo-style)

In leaderless systems, R + W vs N controls consistency:

• N — replicas per item.
• W — write must succeed on this many.
• R — read must reach this many.
• R + W > N guarantees overlap → strong consistency on the latest write.

Cassandra knobs: CONSISTENCY ONE, QUORUM, LOCAL_QUORUM, EACH_QUORUM, ALL.

Tune per-operation: strong reads for billing, weak for activity feed.

Isolation vs Consistency

Easy to confuse:

• Consistency — across nodes, what does a read see?
• Isolation — within one node, how do concurrent transactions interact? (Read uncommitted, read committed, repeatable read, serializable.)

Both matter. A system can be linearisable but read-uncommitted within a transaction, or vice versa. Most relational DBs default to read committed; bump to repeatable-read or serializable when correctness demands.

The Real-World Pattern

Most production systems are mostly eventually consistent with strongly consistent islands:

• Order placement and payment — strongly consistent, linearisable, transactional.
• Order history list — eventually consistent.
• Recommendations — minutes stale.
• Search index — seconds to minutes stale.

The architect's job is to identify which boundaries need strong consistency and which can relax it — and to make the trade-offs visible to product teams.

Practical Heuristics

1. Default to a relational database for the system of record. You get linearisable, ACID transactions for free at the scale most apps need.
2. For multi-region systems, use distributed SQL (Spanner / CockroachDB) when you need strong consistency globally — at higher write latency.
3. For massive-scale, low-latency reads of independently-mutable items, Dynamo-style + tuned quorums are fine.
4. Make staleness budgets explicit: "this read can be 5 seconds stale" should be a documented contract.
5. Don't claim "real-time" without a number attached.

Cert Mapping

The next lesson moves from data-plane consistency to control-plane communication: how services exchange messages without coupling.