Quorum-Based Reads and Writes

During a Black Friday incident, my team's read-after-write consistency guarantee collapsed. A customer would add an item to their cart, see it in the response, then refresh the page and find the cart empty. The root cause was subtle: we had N=3 replicas, W=1 for fast writes, and R=1 for fast reads. The write went to one replica, but the read hit a different replica that hadn't received the replication yet. The customer saw a phantom empty cart.

This is the fundamental tradeoff in distributed storage: you can't have fast writes, fast reads, and strong consistency all at once. The quorum model gives you a dial to control this tradeoff.

The Fundamental Tradeoff

In a distributed system with N replicas of each piece of data, you face a choice on every read and every write:

The quorum model turns this tradeoff into a formula: R + W greater than N. If the sum of the read replica count (R) and write replica count (W) exceeds the total replica count (N), then every read will include at least one node that has the latest write.

The Quorum Model

Defining N, R, and W

The Core Formula: R + W greater than N

When R + W is greater than N, the read set and write set are guaranteed to intersect on at least one node. This means:

Given:
- N replicas
- W replicas acknowledged the latest write
- R replicas are consulted on the read

If R + W > N:
  The intersection of the R read replicas and the W write replicas
  contains at least one node with the latest data.
  => Strong consistency (no stale reads).

Let's verify this with examples:

Walk-Through: N=3, W=2, R=2 (Standard Quorum)

The read consulted A and B, both of which have the latest value. Even though C is stale, it doesn't matter because the read quorum (R=2) didn't need C.

Walk-Through: N=3, W=1, R=1 (No Guarantee)

The read happened to hit node B, which never received the write. Without quorum intersection, stale reads are possible.

Quorum Read Flow Diagram

Consistency Levels in Practice (Cassandra)

Apache Cassandra implements the N, R, W model with named consistency levels. This allows each operation to choose its own tradeoff.

Cassandra Consistency Levels for N=3

Consistency Combinations for N=3

Dynamo's Approach

Amazon's Dynamo paper (2007) defined the modern quorum-based approach to distributed storage. While Cassandra is often called "Dynamo-inspired," there are important differences in how they handle quorum.

Dynamo's Typical Configuration

How Dynamo Processes a Write

Read-Time Reconciliation

Dynamo's read path is unusual: instead of returning the latest value from a single node, it returns ALL versions of a key that are not dominated by any other version (siblings):

Client reads key K with R=2:
  
  Node A returns: K=42, clock=[A=3, B=2, C=1]
  Node B returns: K=99, clock=[A=2, B=3, C=1]
  
  Coordinator compares clocks:
    [A=3, B=2, C=1] vs [A=2, B=3, C=1]
    Neither dominates the other (A: 3 > 2, B: 2 < 3)
    => These updates are CONCURRENT
    => Return BOTH versions to client
  
  Client must reconcile: e.g., merge values or pick one
  Client writes merged result with merged clock

This read-time reconciliation is what makes Dynamo's quorum model "eventually consistent with no data loss." Even if concurrent writes produce conflicting values, the system surfaces both to the client (or application logic) rather than silently dropping one.

Tradeoff Analysis

R=1, W=N: Read-Optimized

R=N, W=1: Write-Optimized

R=W=N/2+1: Balanced (Standard Quorum)

The Availability Tradeoff

Sloppy Quorum

Standard quorum requires writes and reads to reach specific replicas (the "preference list"). During a network partition, some of those replicas may be unreachable. Sloppy quorum relaxes this: the coordinator accepts writes from the first N healthy nodes in the preference list.

How Sloppy Quorum Works

Hinted Handoff

Hinted handoff is the mechanism that makes sloppy quorum eventually consistent:

When Quorum Breaks: Network Partitions

Quorum works in theory, but network partitions create edge cases:

The Write-Read Gap Scenario

Time 0: Client writes X=5 with W=2 (acknowledged by A and B)
Time 1: Network partition separates {A, B} from {C}
Time 2: Client writes X=10 with W=2 (only {B, C} reachable)
         B has X=5 and X=10
         C only has X=10
         A only has X=5
Time 3: Client reads X with R=2
         If read hits {A, C}: A has X=5, C has X=10
         Neither dominates! => CONFLICT

This is where vector clocks (from the previous tutorial) become essential. Quorum guarantees that R and W sets intersect, but intersection doesn't guarantee the latest value on the intersected node—an old version might coexist with a new one on the same node. Vector clocks resolve this by tracking causal history.

The CAP Connection

Real-World Tuning Guidance

Selecting N

Selecting R and W

Production Patterns

Key Takeaways

• The quorum formula R + W > N guarantees read-write intersection for strong consistency
• N defines durability (more replicas = higher fault tolerance but more write amplification)
• R defines read consistency: R=1 is fastest, R=N is slowest but most consistent
• W defines write consistency: W=1 is fastest, W=N is slowest but ensures all replicas are updated
• The balanced configuration (N=3, R=W=2) is the most common production choice
• Sloppy quorum trades strict consistency for availability during network partitions by writing to any N healthy nodes
• Hinted handoff restores consistency after sloppy quorum by delivering writes to their intended replicas when they recover
• R=1, W=N gives fast reads with guaranteed consistency (all replicas have the latest write)
• R=N, W=1 gives fast writes but slow reads and weaker consistency guarantees
• Vector clocks are essential for resolving conflicts when quorum reads discover concurrent updates across replicas
• In production, choose your N, R, W based on your specific latency, consistency, and availability requirements—there is no one-size-fits-all configuration
• The CAP theorem means you must choose: strict quorum (CP) or sloppy quorum (AP) during network partitions

This concludes the Core Algorithms phase. You now understand the foundational algorithms that power distributed systems: consistent hashing, vector clocks, cache eviction, distributed ID generation, and quorum-based reads and writes. These algorithms appear in every major distributed data store, caching system, and ID generation service you'll encounter.