# DDIA Ch. 12: dual writes diverge two stores forever; one ordered CDC log converges them ## Concept Two clients concurrently set the same key X — client 1 wants X=**A**, client 2 wants X=**B** — and each writes to a **database** *and* a **search index** as two separate writes (a *dual write*). Nothing coordinates the two write paths, so the four writes interleave. Line them up (DDIA Figure 12-4) so the database applies A-then-B and ends on **B**, while the index applies B-then-A and ends on **A**. The two stores now **disagree permanently** — `db[X]=B`, `index[X]=A` — and no error was raised. Then route every change through a **single ordered change-data-capture log**: one leader serializes the writes into `[(X,A),(X,B)]`, the db applies the log in order, and the index is a **follower** that replays the *same* log in the *same* order. Both stores converge to **B**. Same two SETs; the only difference is whether there is one agreed order or two independent ones. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 12 — *Stream Processing*, §"Keeping Systems in Sync" / §"Change Data Capture". The claim under test, tightly paraphrased: > With dual writes to two systems, concurrent writes can be applied in different > orders by each system, leaving them permanently inconsistent — whereas change > data capture makes one database the leader and turns the others into followers of > its ordered change log. The book's Figure 12-4 shows two clients' dual writes interleaving so a database and a search index end up disagreeing; Figure 12-5 shows a single ordered change log making one store the source of truth and the others its followers. Here you build the smallest real thing that exhibits both. ## Prerequisites The surprise is about *order*: two write paths with no agreed order vs. one. Each line notes where it shows up. 1. **Dual writes / the two-writes problem** — updating two stores (a db and a derived index) with two independent writes per client, rather than one write the others derive from. It shows up as `db[X]=v; index[X]=v` issued as two separate ops. [DDIA §"Keeping Systems in Sync"; Kleppmann, "Using logs to build a solid data infrastructure" (2015).] 2. **Race conditions & interleaving** — concurrent operations with no enforced order can commit in any interleaving, and some interleavings leave inconsistent state. It shows up as the four writes committing in an order the code lays out explicitly. [Wikipedia, "Race condition § Software".] 3. **Change data capture (CDC)** — capture every change to a database as an ordered stream so other systems can replay it. It shows up as the `[(X,A),(X,B)]` log the db emits and the index consumes. [DDIA §"Change Data Capture"; Debezium docs, "What is change data capture?".] 4. **A log as a single source of truth / total order** — an append-only log fixes one order for all consumers; every derived store is then a deterministic function of that one order. It shows up as the db (leader) and index (follower) replaying the identical log. [Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013).] If only one is new, make it **#4 — one ordered log carries the whole result**: the moment both stores derive from a single total order, divergence is impossible. ## Environment Deterministic, so the values reproduce exactly: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.15.0a8, standard library only — no Docker, no deps - No real concurrency: the interleaving is modeled explicitly as an ordered list of writes, so the result is fixed every run - Fixed workload: client 1 → X=A, client 2 → X=B, into two stores (`db`, `index`) ## Mental model Two stores, `db` and `index`, each a plain dict; two clients, each setting key X. Only the *plumbing between the writes* differs. - **Dual writes** — each client issues two independent, unordered writes: `db[X]=v` and `index[X]=v`. There is no shared order between the db path and the index path, so the four writes (two clients × two stores) can interleave *any* way. Some interleavings let the db settle on one value and the index on another — the two stores are updated by two separate races. - **One ordered CDC log** — the writes are first serialized by a leader into a single append-only log `[(X,A),(X,B)]`. The db applies the log in order to build its state; the index is a *follower* that replays the exact same log in the exact same order. There is now only one write path and one order; every derived store is a deterministic function of that order, so they cannot disagree. Both stores are the same kind of dict; the exercise runs the same two SETs through each design. The whole thing is one script, `code/dual_write_divergence.py`. ## Setup ``` cd study/ddia/ch12/code python3 dual_write_divergence.py ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — dual writes into two stores **Predict.** Client 1 sets X=A and client 2 sets X=B, each writing to both the db and the index as separate writes. Suppose the db commits A then B (so `db[X]=B`). A smart reader expects the index to *also* land on B — both stores just saw the same two writes. What does the index end on, and do the two stores agree? **Observe.** ``` Part 1 -- dual writes: two clients, two stores, four unordered writes --------------------------------------------------------------------- client 1 wants X=A ; client 2 wants X=B each client writes the DB and the INDEX as two separate writes the four writes interleave and commit in this order (Figure 12-4): 1. client 1 -> db [X] = A 2. client 2 -> db [X] = B 3. client 2 -> index[X] = B 4. client 1 -> index[X] = A final state: db[X] = B index[X] = A db != index -> DIVERGED, and nothing raised an error. the database says B, the search index says A -- forever. ``` **`db[X]=B` but `index[X]=A` — they disagree, and nothing threw.** The db saw A-then-B; the index saw B-then-A. Because the two write paths have no shared order, the last write to reach the db (B) is not the last write to reach the index (A). The stores are now permanently inconsistent, with no exception, no log line, no clue. ### Step 2 — one ordered CDC log **Predict.** Now both clients' writes go through a single ordered change log `[(X,A),(X,B)]`: the db applies it in order, and the index is a follower that replays the same log in the same order. What does each store end on — and do they agree this time? **Observe.** ``` Part 2 -- one ordered CDC log: the leader serializes, the index follows ----------------------------------------------------------------------- both clients' writes go through ONE ordered change log (Figure 12-5): offset 1: SET X = A offset 2: SET X = B db replays the log in order -> db[X] = B index replays the SAME log -> index[X] = B db == index == B -> CONVERGED. the index no longer races the db; it just replays the db's order. ``` **Both land on B — converged.** The log fixes one order (A then B) for *everyone*. The index no longer races the db; it just replays the db's order. Because both stores are now the same deterministic function of the same sequence, they cannot end on different values. ## What you should see - Dual writes: `db[X]=B` and `index[X]=A` — the two stores **diverge**, and no error is raised. - One CDC log: `db[X]=B` and `index[X]=B` — the two stores **converge** to B. - Same two SETs (X=A, X=B) both times; only the presence of one agreed order changes the outcome. ## Why Two stores updated by two independent writes have no agreed order between them. When client 1 and client 2 each write to the db *and* the index, the db path and the index path are separate races: the interleaving that reaches the db (A, then B) and the interleaving that reaches the index (B, then A) are decided independently by timing, so the *last* write to win at the db need not be the last write to win at the index. Once that happens the two stores hold different values with equal conviction — there is no write that is "wrong," no constraint violated, nothing to raise. That is why the divergence is silent and permanent: the system did exactly what each of the four writes asked, in an order nobody agreed on. Routing every change through one ordered log removes the *second write path*. The db becomes the leader: it decides one order and emits it as an append-only log. The index stops being an independent writer and becomes a follower — it does not accept its own writes, it only replays the db's log, in the db's order. Now both stores are deterministic functions of the *same* sequence of changes, and two deterministic functions of the same input cannot produce different outputs. The index cannot race the db because it no longer writes independently of it; it can only lag, then catch up to the same state. Convergence isn't a merge that reconciles a conflict after the fact — it's the absence of a conflict, because there was only ever one order. **The boundary — the divergence needs two *independent* write paths *and* concurrency.** Remove either and it vanishes. A single writer (only one client touching X) gives the db and index the same order for free. Wrapping both stores' writes in *one transaction* — a distributed transaction, or the db and index sharing one commit — also forces a single order, at the cost of coupling their availability. CDC picks the third option: keep the writes cheap and asynchronous, but make one store the source of truth and *derive* the rest from its log, so the only order that exists is the one the leader already chose. ### Go deeper 1. **Compact the log and rebuild.** A CDC log can be *log-compacted* (keep only the latest value per key) and used to rebuild a derived store from scratch — feed the compacted log to a fresh, empty index and watch it reconstruct the same converged state. A sibling Ch. 12 exercise. 2. **The outbox pattern.** The db and the log can still diverge if you write them *separately* (the two-writes problem, one level down). The outbox pattern writes the change and its log entry in one local transaction, then ships the log entry — restoring a single source of truth without a distributed transaction.