study › DDIA › Chapter 12 · Stream Processing

Dual Writes Diverge Two Stores Forever; One Ordered CDC Log Converges Them

Two clients concurrently set key X — client 1 wants A, client 2 wants B — each writing to a database and a search index as two separate writes. Interleaved (Figure 12-4), the db ends on B and the index ends on A: the stores disagree forever, and nothing threw. Route every change through a single ordered change-data-capture log (Figure 12-5) and both converge to B.

Concept

Two clients concurrently set the same key X — client 1 wants X=A, client 2 wants X=B — and each writes to a database and a search index as two separate writes (a dual write). Nothing coordinates the two write paths, so the four writes interleave. Line them up (DDIA Figure 12-4) so the database applies A-then-B and ends on B, while the index applies B-then-A and ends on A. The two stores now disagree permanently — db[X]=B, index[X]=A — and no error was raised. Then route every change through a single ordered change-data-capture log: one leader serializes the writes into [(X,A),(X,B)], the db applies the log in order, and the index is a follower that replays the same log in the same order. Both stores converge to B. Same two SETs; the only difference is whether there is one agreed order or two independent ones.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 12 — Stream Processing, §"Keeping Systems in Sync" / §"Change Data Capture". The claim under test, tightly paraphrased:

With dual writes to two systems, concurrent writes can be applied in different orders by each system, leaving them permanently inconsistent — whereas change data capture makes one database the leader and turns the others into followers of its ordered change log.

The book's Figure 12-4 shows two clients' dual writes interleaving so a database and a search index end up disagreeing; Figure 12-5 shows a single ordered change log making one store the source of truth and the others its followers. Here you build the smallest real thing that exhibits both.

Prerequisites

The surprise is about order: two write paths with no agreed order vs. one. Each line notes where it shows up.

  1. Dual writes / the two-writes problem — updating two stores (a db and a derived index) with two independent writes per client, rather than one write the others derive from. It shows up as db[X]=v; index[X]=v issued as two separate ops. [DDIA §"Keeping Systems in Sync"; Kleppmann, "Using logs to build a solid data infrastructure" (2015).]
  2. Race conditions & interleaving — concurrent operations with no enforced order can commit in any interleaving, and some interleavings leave inconsistent state. It shows up as the four writes committing in an order the code lays out explicitly. [Wikipedia, "Race condition § Software".]
  3. Change data capture (CDC) — capture every change to a database as an ordered stream so other systems can replay it. It shows up as the [(X,A),(X,B)] log the db emits and the index consumes. [DDIA §"Change Data Capture"; Debezium docs, "What is change data capture?".]
  4. A log as a single source of truth / total order — an append-only log fixes one order for all consumers; every derived store is then a deterministic function of that one order. It shows up as the db (leader) and index (follower) replaying the identical log. [Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013).]
The one that carries the result If only one prerequisite is new, make it #4 — one ordered log carries the whole result: the moment both stores derive from a single total order, divergence is impossible. Dual writes fail precisely because there are two orders; CDC works precisely because there is one.
Environment these values came from Deterministic, so the values reproduce exactly. Captured on macOS 26.5.2 (Darwin 25.5.0), arm64, Python 3.15.0a8, standard library only — no Docker, no deps. No real concurrency: the interleaving is modeled explicitly as an ordered list of writes, so the result is fixed every run. Fixed workload: client 1 → X=A, client 2 → X=B, into two stores (db, index).

Mental model

Two stores, db and index, each a plain dict; two clients, each setting key X. Only the plumbing between the writes differs.

Both stores are the same kind of dict; the exercise runs the same two SETs through each design. The whole thing is one script, code/dual_write_divergence.py.

Setup

cd study/ddia/ch12/code
python3 dual_write_divergence.py

Do not read the output yet — make each prediction first.

Step 1 · dual writes into two stores

Predict. Client 1 sets X=A and client 2 sets X=B, each writing to both the db and the index as separate writes. Suppose the db commits A then B (so db[X]=B). A smart reader expects the index to also land on B — both stores just saw the same two writes. What does the index end on, and do the two stores agree?

Does the index also end on B?
Part 1 -- dual writes: two clients, two stores, four unordered writes --------------------------------------------------------------------- client 1 wants X=A ; client 2 wants X=B each client writes the DB and the INDEX as two separate writes the four writes interleave and commit in this order (Figure 12-4): 1. client 1 -> db [X] = A 2. client 2 -> db [X] = B 3. client 2 -> index[X] = B 4. client 1 -> index[X] = A final state: db[X] = B index[X] = A db != index -> DIVERGED, and nothing raised an error. the database says B, the search index says A -- forever.

db[X]=B but index[X]=A — they disagree, and nothing threw. The db saw A-then-B; the index saw B-then-A. Because the two write paths have no shared order, the last write to reach the db (B) is not the last write to reach the index (A). The stores are now permanently inconsistent, with no exception, no log line, no clue.

Step 2 · one ordered CDC log

Predict. Now both clients' writes go through a single ordered change log [(X,A),(X,B)]: the db applies it in order, and the index is a follower that replays the same log in the same order. What does each store end on — and do they agree this time?

Do both stores agree this time?
Part 2 -- one ordered CDC log: the leader serializes, the index follows ----------------------------------------------------------------------- both clients' writes go through ONE ordered change log (Figure 12-5): offset 1: SET X = A offset 2: SET X = B db replays the log in order -> db[X] = B index replays the SAME log -> index[X] = B db == index == B -> CONVERGED. the index no longer races the db; it just replays the db's order.

Both land on B — converged. The log fixes one order (A then B) for everyone. The index no longer races the db; it just replays the db's order. Because both stores are now the same deterministic function of the same sequence, they cannot end on different values.

What you should see

Why

Two stores updated by two independent writes have no agreed order between them. When client 1 and client 2 each write to the db and the index, the db path and the index path are separate races: the interleaving that reaches the db (A, then B) and the interleaving that reaches the index (B, then A) are decided independently by timing, so the last write to win at the db need not be the last write to win at the index. Once that happens the two stores hold different values with equal conviction — there is no write that is "wrong," no constraint violated, nothing to raise. That is why the divergence is silent and permanent: the system did exactly what each of the four writes asked, in an order nobody agreed on.

Routing every change through one ordered log removes the second write path. The db becomes the leader: it decides one order and emits it as an append-only log. The index stops being an independent writer and becomes a follower — it does not accept its own writes, it only replays the db's log, in the db's order. Now both stores are deterministic functions of the same sequence of changes, and two deterministic functions of the same input cannot produce different outputs. The index cannot race the db because it no longer writes independently of it; it can only lag, then catch up to the same state. Convergence isn't a merge that reconciles a conflict after the fact — it's the absence of a conflict, because there was only ever one order.

The boundary — the divergence needs two independent write paths and concurrency Remove either and it vanishes. A single writer (only one client touching X) gives the db and index the same order for free. Wrapping both stores' writes in one transaction — a distributed transaction, or the db and index sharing one commit — also forces a single order, at the cost of coupling their availability. CDC picks the third option: keep the writes cheap and asynchronous, but make one store the source of truth and derive the rest from its log, so the only order that exists is the one the leader already chose.

Go deeper

Sources: DDIA 2e, Ch. 12, §"Keeping Systems in Sync", §"Change Data Capture" (Figures 12-4 and 12-5) · Kreps, "The Log" (2013) · Kleppmann, "Using logs to build a solid data infrastructure" (2015).

Source of truth for this exercise: dual-write-divergence.md. Every line above was captured from a real run of code/dual_write_divergence.py on macOS 26.5.2 / arm64 / Python 3.15.0a8, before publishing. The interleaving is modeled explicitly and deterministic, so the values reproduce exactly.