# DDIA Ch. 6: last-write-wins throws away the write that actually happened last ## Concept Multi-leader and leaderless replication let two nodes accept a write to the same key at the same time, so conflicts are inevitable and something has to resolve them. Last-Write-Wins (LWW) is the simplest rule: stamp every write with a timestamp and keep the one with the greatest timestamp. No coordination, no merge — and silent data loss. The "timestamp" is a wall-clock reading, and wall clocks drift. If one node's clock runs fast, a write it made *earlier* in real time carries a *larger* timestamp than a write another node made *later*; LWW keeps the larger timestamp and discards the write that truly happened last — with no error. You will predict the winner, then watch **"C" (the real last write) get silently dropped** while an older "B" survives, purely because a clock was 30 seconds fast. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 6 — *Replication*, §"Last write wins (discarding concurrent writes)": > the name "last write wins" is misleading The book stresses that LWW's surviving write was not necessarily the last one: because timestamps come from clocks that disagree, "last" is whatever clock said the biggest number. This exercise makes that discarded write visible. ## Prerequisites The surprise is that a conflict resolver can be *deterministic and wrong* at the same time. These help; each line notes where it shows up. 1. **Conflict resolution in multi-leader/leaderless replication** — two replicas accept writes to one key independently, so on sync one write must be chosen; both scenarios below are exactly that two-write conflict. 2. **Last-Write-Wins (LWW)** — the rule "keep the write with the greatest timestamp"; it is what `lww_resolve` computes and what picks the winner in each scenario. 3. **Clock skew / unreliable wall clocks** — different nodes' time-of-day clocks disagree by seconds; modeled here as a per-node `offset` added to real time to produce the write's timestamp. 4. **Real order vs. timestamp order** — the write that happened last in wall-clock time need not have the largest timestamp; the transcript prints both orders side by side so you can see them diverge. ### Where to learn them - **LWW and its data loss (#1, #2):** DDIA §"Last write wins (discarding concurrent writes)" and §"Handling write conflicts". - **Why clocks can't be trusted for ordering (#3, #4):** DDIA Ch. 9, §"Unreliable clocks" and §"Timestamps for ordering events" — LWW's failure is a special case of "don't order events by wall-clock time." If only one is new, make it #3 — the whole failure rides on clocks disagreeing. ## Environment Deterministic — no real clocks and no randomness, so the winner reproduces exactly: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.15.0a8, standard library only — no Docker, no third-party packages - Two leaders write `title` to the same key; real times and per-node clock offsets are passed in explicitly, so `timestamp = real_time + offset` is fixed ## Mental model Two leaders, one key `title`, and one conflict resolved by LWW: - **A write** carries `(node, value, real_time, offset)`. Its timestamp is the node's *reading* of the clock: `ts = real_time + offset`. A skewed clock has a nonzero offset. - **Real order** sorts writes by `real_time` — the truth of what happened when. - **Timestamp order** sorts by `ts` — what LWW actually compares. - **LWW** returns the write with the greatest `ts` (ties broken by value so every replica picks the same winner). It never looks at `real_time`; it can't. When the two orders agree, LWW is correct. When a clock skew flips them, LWW keeps an older write and the real-last write is lost silently. The whole exercise is one script, `code/lww_clock_skew.py`. ## Setup ``` cd study/ddia/ch06/code python3 lww_clock_skew.py ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — predict the winner under a skewed clock **Predict.** At real time t=100, node A (clock +30s fast) writes `title="B"`. Then *later*, at t=110, node B (clock accurate) writes `title="C"`. The real-last write is "C". Which value does LWW keep — "B" or "C"? Write down the winner and the two timestamps. **Observe.** ``` Scenario 1 -- node A's clock is skewed +30s (fast) -------------------------------------------------- real (wall-clock) order of events -- what actually happened: t=100s real node A (clock offset +30s) writes title='B' -> stamps ts=130 t=110s real node B (clock offset +0s) writes title='C' -> stamps ts=110 => the write that ACTUALLY happened last: title='C' (node B, t=110s real) timestamp order -- what LWW compares (greatest ts wins): ts=110 title='C' (node B) ts=130 title='B' (node A) => LWW WINNER: title='B' (node A, ts=130) - title='C' (node B, ts=110): LOST (silently discarded) <-- this was the REAL last write! verdict: WRONG: LWW kept an OLDER write; the real-last write was lost with no error. ``` A's write happened **first** in real time but carries the **bigger** timestamp (130 > 110) because A's clock was 30s ahead. LWW keeps ts=130 → "B". The write that actually happened last, "C", is discarded — no conflict raised, no error logged. The data is just gone. ### Step 2 — the control: sync the clocks, keep the same two writes **Predict.** Same two writes, same real times, but now both clocks are accurate (offset 0). Does LWW still lose "C"? **Observe.** ``` Scenario 2 (control) -- clocks synced (both offset 0) ----------------------------------------------------- real (wall-clock) order of events -- what actually happened: t=100s real node A (clock offset +0s) writes title='B' -> stamps ts=100 t=110s real node B (clock offset +0s) writes title='C' -> stamps ts=110 => the write that ACTUALLY happened last: title='C' (node B, t=110s real) timestamp order -- what LWW compares (greatest ts wins): ts=100 title='B' (node A) ts=110 title='C' (node B) => LWW WINNER: title='C' (node B, ts=110) - title='B' (node A, ts=100): LOST (silently discarded) verdict: CORRECT: LWW kept the real-last write. ``` With clocks agreeing, timestamp order equals real order (100 < 110), so LWW keeps "C" — the actual last write. The algorithm did not change; only the clocks did. LWW is correct exactly when clocks agree, and the bottom line makes it explicit: ``` bottom line: skewed clock -> LWW correct? False (the real-last write 'C' was silently lost) synced clock -> LWW correct? True (same two writes, but now 'C' survives) LWW is only as trustworthy as the clocks agree. "Last write wins" is a misnomer: the survivor is the greatest-timestamp write, not the last one. ``` ## What you should see - Under +30s skew, **LWW keeps "B" and silently discards "C"** — the write that actually happened last — with `verdict: WRONG` and no error. - With synced clocks, the **same two writes** resolve to "C" (`verdict: CORRECT`) — LWW's correctness depends entirely on the clocks, not the writes. - "Last write wins" is a misnomer: the survivor is the greatest-*timestamp* write, which need not be the last one. ## Why LWW's only input is the timestamp, and the timestamp is `real_time + offset` — the node's own clock reading. LWW picks `max(ts)`. When both offsets are 0, `ts` is a monotonic function of `real_time`, so `max(ts)` *is* the last write: the control scenario keeps "C" correctly. The algorithm is not broken; it is exactly right whenever the clocks form a shared, consistent order of time. The skew breaks that assumption. A's `+30s` offset adds a constant to A's timestamps only, so a write A made 10 seconds *before* B's still reads `100 + 30 = 130 > 110`. LWW compares 130 vs 110 and keeps A's older "B". It has no way to know 130 came from a fast clock rather than from a later moment — a timestamp carries no evidence of the clock that produced it. So the resolution is deterministic and repeatable (re-run it, "C" is always lost) yet causally wrong. And because a conflict "resolved" is a conflict *silenced*, nothing surfaces: no exception, no divergence alarm, no second version to reconcile. "C" simply never existed as far as the surviving replica is concerned. **The boundary — LWW trades durability for convergence, and clocks set the price.** LWW's real guarantee is only that every replica eventually agrees on *some* value, not that it keeps the *right* one; the moment two nodes' clocks disagree by more than the gap between two writes, "the right one" and "the one LWW keeps" can differ, and a write vanishes. The fix is to stop ordering events by wall-clock time: use version vectors / causal tracking to detect genuine concurrency and keep *both* conflicting writes (siblings) for the application to merge, or logical clocks (Lamport timestamps) that at least give a consistent total order without pretending to be real time. LWW is acceptable only when losing concurrent writes is genuinely fine — a cache, a last-seen presence timestamp — and never when each write is data you can't afford to drop. ### Go deeper 1. **Find the skew that flips it.** The failure needs A's offset to exceed the 10s real-time gap. Re-run with offset `+5` instead of `+30` and predict the winner; then `+11`. Where exactly does "C" start surviving? 2. **Keep both writes instead.** Replace `lww_resolve` with a resolver that returns *both* values as siblings whenever their real times differ by less than the clock uncertainty. Predict what the skewed scenario now reports — and what new burden that puts on the application. 3. **Lamport timestamps.** Swap wall-clock `ts` for a Lamport logical clock (counter, node-id tiebreak) that increments on each event a node sees. Predict whether it recovers the *causal* order here, and what it still can't tell you about two truly concurrent writes.