# DDIA Ch. 12: at-least-once delivery double-counts a sum; stamping the offset makes it exact ## Concept A consumer reads an append-only log of 100 "+1" events and folds each one into an external `total`. It commits its consumer offset only *periodically* — a checkpoint every 10 events — so the offset it has durably committed lags the events it has already applied. Crash it in that gap: it has **applied through offset 49** but its last committed checkpoint is at **offset 39**. On restart it resumes from 40 and **reprocesses offsets 40–49** — at-least-once delivery, so those ten events arrive a second time. The naive consumer just increments, so the replay lands the counter at **110, not 100** — redelivered messages *inflate* it. Stamp the triggering offset alongside the value and reject anything not strictly newer, and the same replay is a no-op: the total is exactly **100**. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 12 — *Stream Processing*, §"Fault tolerance" (§"Idempotence", §"Exactly-once execution"): > An idempotent operation is one that you can perform multiple times, and it has > the same effect as if you performed it only once. [...] you can safely retry > without causing the operation to take effect twice. The book's framing: a stream processor that must survive crashes can cheaply guarantee *at-least-once* delivery (replay from the last committed offset), but not exactly-once without more work. Making the side effect **idempotent** — often by recording which offset produced it — turns at-least-once into *effectively-once*. Here you watch the double-count happen, then close it with one offset stamp. ## Prerequisites The surprise lives in the gap between two separate writes — the side effect and the offset commit. These help; each line notes where it shows up. 1. **Consumer offsets & checkpointing** — a log consumer tracks how far it has read as an *offset*, and commits it periodically rather than per-event (committing every event is expensive). It shows up as the checkpoint that fires every 10 events and lags the applied work at crash time. 2. **At-least-once vs exactly-once** — on restart a consumer resumes from its last *committed* offset, so anything applied after that offset but before the commit is delivered again. At-least-once = "every event applied ≥ 1 time"; the events between offset 40 and 49 are applied twice. 3. **Idempotent operations** — an operation whose second application has no additional effect. A bare `total += 1` is *not* idempotent; a "set the value and remember the offset" write *is*. It shows up as the `IdempotentStore.apply` that a replay leaves unchanged. 4. **Deduplication by offset/key** — recording the highest offset already folded in lets the store reject a redelivered event as a duplicate. It shows up as the `offset <= highest_applied` skip that drops offsets 40–49 on replay. ### Where to learn the prerequisites - **Offsets, at-least-once, exactly-once (#1–#2):** DDIA §"Fault tolerance" and §"Exactly-once execution"; Kafka's docs on consumer offsets and delivery semantics for how commit cadence creates the replay window. - **Idempotence & dedup (#3–#4):** DDIA §"Idempotence" — the operation-id / offset-stamping technique, and why the effect and the offset must be seen together for the dedup to be sound. If only one is new, make it **#3 idempotence-via-offset** — recording *which* offset produced a value is the single move that carries the result: it converts a non-idempotent increment into an idempotent set-if-newer, and at-least-once into effectively-once. ## Environment Deterministic, so the totals reproduce exactly: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.15.0a8, standard library only — no Docker, no deps - No real broker or crash: the log, the periodic checkpoint, the crash point, and the replay are all modeled in one process, so the result is fixed every run - Fixed workload: 100 "+1" events (offsets 0..99); crash applied-through 49, committed 39 → replay of 40..49; true total 100 ## Mental model At-least-once delivery means *every event is applied at least once* — never lost, but possibly repeated. A stream consumer earns that guarantee cheaply: it commits its read position (the offset) only now and then, and on restart it rewinds to the last committed offset and replays forward. Everything it had already applied past that offset gets applied again. - **The side effect and the offset commit are two separate writes.** The consumer first applies an event to the external total, and *separately* commits its offset. A crash can fall between them — apply happened, commit didn't — so recovery replays an effect that already took hold. - **A non-idempotent effect turns the replay into a double-count.** `total += 1` has no memory; applied twice, it adds twice. Ten replayed events add ten too many. - **Idempotence makes the re-apply a no-op.** Store the *highest applied offset* next to the value and only apply an event whose offset is strictly newer. The replayed offsets 40–49 are ≤ the stored 49, so they are rejected — the second delivery has the same effect as none. At-least-once becomes effectively-once. Both stores implement the same `apply(offset, delta)` interface; the exercise drives each through the identical crash-and-replay schedule. The whole thing is one script, `code/at_least_once_double_count.py`. ## Setup ``` cd study/ddia/ch12 python3 code/at_least_once_double_count.py ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — the workload **Predict.** The log has 100 events, each "+1". If every event is applied exactly once, what does the external total read? **Observe.** ``` An append-only log of 100 "+1" events, offsets 0..99. A consumer applies each event to an external total, checkpointing its offset every 10 events. It crashes having applied through offset 49 with its offset last committed at 39 -- so restart replays 40..49. every event is "+1", so the true total is 100. ``` 100 — no surprise yet. The surprise is what a crashed-and-replayed consumer does to it. ### Step 2 — the naive consumer crashes and replays **Predict.** The consumer applied through offset 49 but committed its offset only up to 39, then crashed. It restarts, resumes from offset 40, and reprocesses 40–49 before finishing 50–99 — incrementing on every delivery, with no dedup. What is the final total, and by how much is it off? **Observe.** ``` NAIVE increment (at-least-once, no dedup): first run: applied offsets 0..49, total = 50, last checkpoint committed = offset 39 *** CRASH *** (applied through 49, but offset only committed up to 39) restart: resume from committed offset 40 -> redelivers offsets 40..49 (10 events), then 50..99 skipped 0 offsets -- every redelivered event was applied again final total = 110 (true total = 100, INFLATED by 10) ``` **110, not 100 — inflated by exactly 10.** The ten events between the last committed offset (39) and the crash point (49) were already applied in the first run; the replay applies them a second time. A lost message would undercount; a *redelivered* one overcounts. Either way the counter is wrong. ### Step 3 — stamp the offset and dedup **Predict.** Same log, same crash, same replay of 40–49 — but now the store records the highest offset it has applied and refuses any event whose offset is not strictly newer. What is the final total, and which offsets does it skip? **Observe.** ``` IDEMPOTENT set-if-newer (dedup by offset): first run: applied offsets 0..49, total = 50, last checkpoint committed = offset 39 *** CRASH *** (applied through 49, but offset only committed up to 39) restart: resume from committed offset 40 -> redelivers offsets 40..49 (10 events), then 50..99 skipped 10 already-applied offsets: 40..49 final total = 100 (true total = 100, exactly the number of events) ``` **Exactly 100 — the ten replayed offsets are skipped.** The first run left the store remembering `highest_applied = 49`. On replay, offsets 40–49 are all ≤ 49, so `apply` returns without touching the total; the first strictly-newer offset it acts on is 50. The redelivery still happens — at-least-once is unchanged — but its effect is now a no-op. ### Step 4 — the summary line **Predict.** Side by side, what do the naive and idempotent consumers report, and what accounts for the difference? **Observe.** ``` summary: naive -> 110 (+10 from the 10 replayed events) idempotent -> 100 (replayed events rejected as duplicates -> effectively-once) ``` **One difference: whether the store remembers which offset it already applied.** Same delivery guarantee, same replay — the naive store has no memory and double-counts, the idempotent store stamps the offset and dedups. ## What you should see - The true total is **100** — 100 events, each "+1". - The naive consumer ends at **110**, inflated by **10** — the replayed offsets **40–49** applied twice. - The idempotent consumer ends at exactly **100**, **skipping offsets 40–49** on replay. - Both consumers ran the *identical* crash-and-replay schedule; only the presence of the offset stamp differs. ## Why At-least-once delivery is not a bug — it is the honest guarantee a crash-tolerant consumer can afford. Committing the read offset after *every* event would be correct but expensive, so real consumers commit periodically and accept that a crash rewinds them to the last committed offset. The cost of that cheap guarantee is a replay window: every event applied after the last commit but before the crash will be delivered again. The double-count comes from a subtler place: **the side effect and the offset commit are two separate writes, and a crash can split them.** The consumer applies event 49 to the external total, then — as a distinct operation — commits its offset. Between those two writes the state is inconsistent: the effect has happened but the log still believes it hasn't. Crash there and recovery faithfully replays from offset 40, re-applying effects that already took hold. With a non-idempotent operation (`total += 1`, which has no notion of "already done"), each replayed delivery adds again — ten replays, ten spurious increments, total 110. This is why the book warns that partial failure between the effect and the offset commit corrupts derived state. Recording **which** offset produced a value is what closes the gap. The idempotent store keeps `highest_applied` next to `total` and turns the increment into a *set-if-newer*: apply only when `offset > highest_applied`, and stamp the new offset atomically with the value. Now a redelivered event carries its own offset (40..49), the store sees each is ≤ the 49 it already recorded, and rejects it — the second application has the same effect as none, which is precisely the definition of idempotence. At-least-once delivery is unchanged; what changed is that a repeat is now harmless, so the guarantee is *effectively-once*. ### The boundary — you need a non-idempotent effect *and* a non-atomic checkpoint The double-count needs both ingredients, and removing either one dissolves it. If the side effect and the offset commit happen in **one atomic transaction** (write the total and the committed offset to the same store in a single commit — a transactional sink, or the transactional-outbox pattern), a crash can't split them: either both landed or neither did, so there is nothing to replay past. And if the operation is **naturally idempotent** (set a key to a value, insert with a unique key, `max` a gauge), a second delivery is already a no-op and needs no offset stamp at all. The inflation you saw requires the unlucky combination: an effect that accumulates (so repeats matter) committed *separately* from the offset (so a crash can replay it). Stamping the offset is the cheap fix when you can't get atomicity across the effect and the log. ### Go deeper 1. **Make the checkpoint atomic instead.** Store `total` and the committed offset in the same object and commit them together; confirm the naive increment now survives the crash without a dedup set, because recovery never replays a committed effect. 2. **Two-phase commit to the sink.** Have the consumer prepare the write and the offset commit across the log and the external store, and commit both or neither — the distributed-transaction route to exactly-once, and the coordination price it charges (compare with Kafka's exactly-once transactions). 3. **The dual-write problem.** Write the same fact to two systems (the total *and* a search index) without a shared transaction and watch them diverge under a crash — a sibling Ch. 12 exercise on why "just write to both" isn't exactly-once either.