study

Books you read, turned into labs you run. Each exercise makes you predict a specific behavior, run the real thing, and check your prediction against a real transcript — the gap is where the mental model gets built.

Designing Data-Intensive Applications

Kleppmann & Riccomini, 2nd ed. (2025).

Chapter 2 · Defining Nonfunctional Requirements

• Tail-Latency Amplification — a backend call slow just 1% of the time, fanned out to 100 parallel calls, makes 63% of end-user requests slow. Predict "still ~99% fast"; measure ~37%.
• The Mean Hides the Tail — for right-skewed latency, 69% of requests are faster than "average," yet p99 is 6× the mean. The average overstates the typical request and understates the tail at once.
• Averaging Percentiles Is a Lie — average ten servers' p99s and you land 21% below the true pooled p99. Predict "close enough"; measure the gap.

Chapter 3 · Data Models and Query Languages

• ADD COLUMN Is Instant, Until It Isn't — adding a NOT NULL column with a default to a 10M-row Postgres table takes 0.8 ms, not the full rewrite everyone predicts — but a volatile default rewrites it in ~8 s.
• Reachability Needs Recursion — "is Boise in North America?" A two-level join silently answers no (it's 4 hops deep); WITH RECURSIVE gets it right at any depth. A fixed join count can't express transitive reachability.
• One Company, Two Names — a denormalized rename touches 666,666 rows vs 1; interrupt it and one company splits into two names — an inconsistency the normalized schema can't represent.
• The Document Wins the Whole Read and Loses the Field — a JSONB document reads ~20× fewer pages than the join for a whole profile, but ~12× more for a single field. Locality cuts both ways.
• You Can Index Inside a Document — a nested-field filter scans 2M rows (~98 ms); a GIN index makes it ~27× faster — with the catch that it serves @>, not ->> equality.
• The Ratings Matrix That's 99% Empty — a 1%-full ratings matrix is 200 MB dense vs 4 MB sparse (50×), following 1/(2×density). Netflix-scale dense is 34 GB.

Chapter 4 · Storage and Retrieval

• The Column Store Reads Only What It Needs — a 2-column query over a 30-column table makes a row store read 100% of the data and a column store 0.6% — 165× less — from layout alone.
• Sorting Rows Changes How Big They Are — a column store is 5.5× smaller than the row store; reordering the same rows shrinks it another 1.4×, one column collapsing 30×. Sort order is a compression lever.
• Ten Bits per Key Buys a 1% Miss Rate — a Bloom filter needs ~10 bits (not bytes) per key for a 0.8% false-positive rate, matching the formula — and its size is independent of key length.
• The Same Rows Cost 4× More in Random Order — load 10M rows into a B-tree in random key order and it's 4× slower with a 31% bigger index (69% vs 90% leaf fill). Why LSM writes sequentially.
• The Covering Index That Still Reads the Heap — a covering index keeps reading 1,637 heap blocks until one VACUUM sets the visibility map; then heap fetches drop to zero.
• LIKE Reads Every Row; an Inverted Index Reads Five Buffers — finding a word with ILIKE '%…%' scans 5M rows (~1.25 s); a tsvector+GIN index does it in ~19 ms, the probe touching 5 buffers — ~65× faster.
• Storing 230 MB Wrote 2 GB to Disk — an LSM (RocksDB) writes 2,091 MB to durably store 230 MB — a 9× write amplification, ~7× of it compaction — while the database ends up just 235 MB.

Chapter 5 · Encoding and Evolution

• The Same Record, 81 Bytes to 32 — one record encoded five ways: JSON 81 → MessagePack 66 → Protobuf 33 → Avro 32 bytes. Each rung drops a layer of self-description; 38% of the JSON is field names.
• Compression Closes the Gap — Avro is 2.7× smaller than JSON raw, but only 1.4× after gzip — the binary size win largely collapses once you compress a batch.
• The ID That Changed on the Way In — a 64-bit ID through a double-based JSON parser comes back off by 21, no error; a typed Protobuf int64 round-trips it exactly.
• Schema Evolution: Old Code Reads New Data, New Code Reads Old — add a field and old and new readers still interoperate (forward + backward compatible) — until you add one without a default, which breaks old data.
• The Field-Tag Landmine: Reuse a Tag, Silently Corrupt — reuse a Protobuf tag number and old bytes decode as the wrong field — a timestamp silently read as a user id, no error. A fresh tag is the only safe move.
• Varint & Zigzag — −1 Costs 10 Bytes, or 1 — a negative int64 sign-extends to 10 bytes; sint64 zigzag makes it 1. A batch of small negatives is 7.3× larger as int64.
• Avro Is the Smallest — and Unreadable Without Its Schema — Avro (21 bytes) beats Protobuf (24) by dropping tags, but decode with the wrong schema and it errors; Protobuf's tags survive a partial, renamed schema.

Chapter 6 · Replication

• Your Own Write, Gone From the Replica — write to the leader, read an async replica, and your own comment comes back missing (count 0) for ~5 s; reading the leader is the fix.
• Refresh the Page, and the Comment Is Gone — two replicas at different lag make a user's reads run backward in time (1 then 0); sticky routing gives monotonic reads.
• The Same SQL, Two Different Rows — a follower replaying INSERT … now(), random() gets different values than the leader — why databases ship rows/WAL, not statements.
• Last-Write-Wins Discards the Write That Actually Happened Last — under a skewed clock, LWW keeps an older write and silently drops the one that truly happened last — no error, data gone.
• Quorums Overlap by w + r > n — And Still Read Stale — w + r > n forces the read and write sets to overlap (9/9 combos), yet a sloppy quorum still returns stale.
• Version Numbers Catch the Conflict LWW Hides — two concurrent writes: LWW keeps [eggs] and loses milk; version numbers keep both as siblings, losing nothing.
• A G-Counter Converges; a Naive Counter Loses 80 Increments — a naive merge keeps 100 of 180 increments (loses 44%); a G-Counter CRDT converges to the exact 180 under any merge order.

Chapter 7 · Sharding

• hash % N Is a Rebalancing Catastrophe — adding one node to ten moves 91% of all keys under hash % N; a fixed 1,000-shard layer moves just 9% — 10× less data over the wire, from indirection alone.
• Consistent Hashing Moves Only a Fair Share — rendezvous hashing moves 9% of keys and zero between nodes that didn't change; hash % N moves 91%, 163k of them pure churn between unchanged nodes.
• A Perfect Hash, and One Shard Still Melts — a good hash spreads keys evenly (1.06×), yet a Zipf celebrity key drives 19% of the load onto its one shard; splitting it helps but Zipf's head bites back (2.13×).
• Time-Ordered Keys Funnel Every Write Into One Shard — a timestamp partition key sends 100% of writes to the newest shard while 11 sit idle; a sensor-ID prefix spreads them to 1.02× the mean — at the cost of scattering the range query.
• Hashing the Key Destroys Range Queries — a 1,000-key range scan touches 1 shard under key-range sharding but all 16 under hash sharding; a composite key wins the range back.
• Local vs Global Secondary Indexes — color=red reads 8 shards locally vs 1 globally; writing one car touches 1 shard locally vs 3 globally. You choose which side pays.
• Why Cassandra Uses 16 Vnodes and ScyllaDB 256 — one random range per node leaves the busiest 2.8× over fair share; 16 vnodes cut it to 1.4×, 256 to 1.1× — the spread shrinking like 1/√V.

Chapter 8 · Transactions

• Postgres Won't Dirty-Read Even When You Beg It To — set READ UNCOMMITTED (Postgres accepts it by name) and a concurrent session still reads the old committed value, never the uncommitted one — MVCC makes the dirty read impossible.
• Read Skew Makes $100 Vanish — Until REPEATABLE READ — under READ COMMITTED, two reads in one transaction sum to $900; REPEATABLE READ's single snapshot restores the true $1,000.
• The Lost Update: Two Increments, Final Value Off by One — two read-modify-write increments leave 43, not 44; an atomic value = value + 1 or FOR UPDATE fixes it, and REPEATABLE READ aborts one with 40001.
• Two On-Call Doctors Both Leave, and "Repeatable Read" Lets Them — both see 2 on call, both go off → zero on call; snapshot isolation permits it, only SERIALIZABLE aborts one with 40001.
• A Phantom Double-Books a Room, and FOR UPDATE Can't Stop It — both check the room is free (0) and both book it; FOR UPDATE can't lock a row that doesn't exist yet — SERIALIZABLE (40001) or an exclusion constraint (23P01) closes it.
• Under MVCC a Writer Doesn't Wait for a Reader — a writer commits in ~1 ms alongside an open reader that keeps seeing its snapshot; add one SELECT … FOR UPDATE and the writer blocks ~1 s.
• Two Transactions Deadlock — Postgres Shoots One, Doesn't Hang — opposite lock order deadlocks; after ~1 s the detector aborts a victim with 40P01 and the survivor commits. Consistent lock ordering prevents it.
• Compare-and-Set: the Loser Silently Gets UPDATE 0 — two guarded writes race; the loser's WHERE matches nothing → UPDATE 0 (a lost update prevented lock-free); at REPEATABLE READ it aborts with 40001 instead.
• One Row, Updated 100,000 Times — 8 kB or 2.75 MB? — an UPDATE is a delete + insert; 100k updates keep the HOT (unindexed) table flat at 8 kB but bloat the indexed one 344× to 2.75 MB, until VACUUM reclaims it.
• A Prepared Transaction Holds Its Lock Through a Full Server Restart — PREPARE TRANSACTION leaves an in-doubt txn that keeps its row lock and survives a restart in pg_prepared_xacts; only COMMIT/ROLLBACK PREPARED frees it.

Chapter 9 · The Trouble with Distributed Systems

• A Fencing Token Bounces a Zombie Leaseholder's Write — a paused leaseholder wakes, still thinks it's leader, and writes — corrupting storage. A monotonic fencing token the resource checks (33 < 34) makes the stale write bounce.
• There Is No Safe Failure-Detection Timeout — a "generous" 6×-median timeout still declares a healthy node dead 3.66% of the time; driving false positives to zero pushes detection delay toward infinity. A heavy tail has no safe cutoff.
• Commit-Wait: Fix an Uncertain Clock by Deliberately Waiting — a point timestamp from an uncertain clock orders causal writes backwards 38.9% of the time; Spanner's commit-wait (pause 2ε) drops it to 0 — the fix is going slower on purpose.
• Measuring a Duration With the Wall Clock Goes Negative — an NTP step-back mid-measurement makes wall-clock elapsed −3.000 s (a lease that never expires); the monotonic clock ignores the step and reads the true +2.000 s.

Chapter 10 · Consistency and Consensus

• A Linearizability Checker Rejects a History That "Looks Consistent" — every read returns a value that was really written, no op-by-op contradiction — yet a checker flags it not linearizable: once A read the new value, a later B may not read the old one.
• A Lamport Clock Inverts a Cross-Shard "Then", Leaking a Private Photo — set-private (ts 8) then upload (ts 2): with no message between shards there's no causal edge, so a snapshot read leaks the photo. A linearizable ID generator closes it.
• CAP Made Countable: a Partition Costs 12 Refusals or 3 Conflicts — the same partition makes the CP deployment refuse 12 of 20 requests and the AP one diverge on 3 keys. You don't pick 2 of 3; you pick which cost to pay.
• SERIALIZABLE Still Returns a Stale Read — It Is Not Linearizable — under SERIALIZABLE, a transaction's second read still returns 0 after another commits 1, and commits with no error — serializable order isn't real-time order.
• Only the Majority Side of a Partition Can Decide — split 5 nodes 3–2 and the minority just stops; and since every decision needs a majority round-trip, adding nodes raises fault tolerance but not throughput.
• A Backward NTP Step Reorders Timestamps; an HLC Stays Monotonic — a −100 ms step drops a later event's physical ts to 940 after 1030 (effects before causes); a hybrid logical clock absorbs it into a counter and stays ordered.
• A Lamport Timestamp Orders Two Concurrent Events; a Vector Clock Refuses — L(e9)=3 < 4=L(e4) tempts "e9 before e4", but the vector clocks are incomparable — genuinely concurrent. One integer can't tell order from coincidence.

Chapter 11 · Batch Processing

• An Aggregation's Memory Tracks Distinct Keys, Not Input Size — 2,000,000 identical lines cost a hash table one dict entry; hold the line count fixed and its memory swings 7.5× with distinct-key count while sort barely moves.
• A Broadcast Join Moves 124× Less Data Than a Reduce-Side Join — a reduce-side join shuffles both tables (60.5 MB); broadcasting the small side moves 500 KB for the identical result — 124× less network.
• One Hot Key Sends 85% of a Join to a Single Reducer — a celebrity key pins 86.9% of the job on one reducer; 8× the reducers buys ~1× the speedup. Salting the key cuts it to 17.9%.
• Materializing Between Stages Blocks the First Output — MapReduce writes a 3×-replicated intermediate and needs all 100k records before its first output; a pipelined dataflow emits after 101 — a 990× gap.
• Loading a Batch Job's Output Row-by-Row Is ~900× Slower — committing one row at a time takes ~28 s for 50k rows vs 0.03 s bulk (~900×), and a crashed retry duplicates rows that bulk-then-swap never leaks.

Chapter 12 · Stream Processing

• Windowing by Processing Time Invents a 61× Spike — a steady 1 event/second, bucketed by processing time after a 60 s consumer outage, shows a 61× spike that never happened; event-time buckets stay flat.
• A Straggler Silently Undercounts a Fired Window — a window fires at count 5; a late event for it is dropped with no error (published 5, true 6). A watermark + allowed-lateness emits the correction to 6.
• One Stream, One "5-Minute" Width, Four Different Groupings — two events 4m33s apart: a sliding window groups them, tumbling splits them, hopping shares none, session groups them. Alignment, not duration, decides.
• At-Least-Once Delivery Double-Counts a Sum — a crash between applying an event and checkpointing its offset replays 10 events → total 110, not 100; stamping the offset and skipping duplicates fixes it to 100.
• Dual Writes Diverge Two Stores Forever — interleaved writes leave the DB on B and the index on A with no error; routing both through one ordered CDC log converges them.
• A Compacted Changelog Rebuilds the Exact Database From 0.1% of the Records — 1,000,000 writes compact to 937 records (one per live key, 1,067×); replaying only those reconstructs the database exactly.
• A Naive Stream-Table Join Time-Travels — Replay Changes the Answer — a current-state tax lookup bills old sales at today's rate: $100 today but $80 if replayed last year; a point-in-time join gives a stable $94.
• Load Balancing Plus Redelivery Reorders an "In-Order" Stream — a stalled-then-redelivered message completes as m1, m2, m4, m3, m5 across two consumers; a single partitioned log preserves the order (but head-of-line blocks).

Chapter 13 · A Philosophy of Streaming Systems

• Retrying an "Atomic" Transfer Charges $22, Not $11 — atomicity isn't idempotence: a retried transfer moves $22, no error. A client-generated request ID with a UNIQUE constraint makes the retry a no-op — the end-to-end argument.
• A Multishard Transfer Stays Exactly-Once Through a Crash — Without 2PC — atomically append one request, derive both shard effects deterministically, dedup by ID: the money invariant (sum 200) survives a mid-transfer crash; drop the dedup and the payee is double-credited (211).
• Uniqueness Needs a Total Order; Async Multi-Leader Grants 'alice' Twice — two independent leaders each find alice free and both grant it (2 owners); routing claims through one totally-ordered partition grants exactly 1.
• Losing Causal Order Across Two Logs Notifies the Ex You Unfriended — unfriend-then-post across two independent logs arrives reordered, so the notification fires anyway; one ordered partition, or a causal-version reference, suppresses it.

Chapter 14 · Doing the Right Thing

• Dropping "Race" Doesn't Debias the Model — a Proxy Relearns It — "fairness through unawareness" fails: the disparate-impact ratio barely moves (0.29 → 0.44, still failing the 0.80 rule) when you drop the protected attribute, because a neighborhood proxy reconstructs it (AUC 0.96). Drop the proxy too and it reaches 1.03.

Source: github.com/atharh/study · each exercise's .md is the source of truth; the .html is its rendered companion.