# DDIA Ch. 4: storing 230 MB wrote 2 GB to disk ## Concept One logical write becomes many physical writes. An LSM-tree logs each value to its write-ahead log, writes it again when the memtable is flushed to an SSTable, and writes it *again* every time compaction rewrites the SSTable it lives in as the data migrates down the levels. The book calls the ratio — total bytes written to disk over the bytes a plain append-only log would write — **write amplification**. You will store ~230 MB of data in a real LSM engine (RocksDB) and watch it write **2,091 MB** to disk to do it: a **9×** amplification, most of it compaction rewriting the same data over and over. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 4 — *Storage and Retrieval*, §"Write amplification": > With LSM-trees, a value is first written to the log for durability, then again > when the memtable is written to disk, and again every time the key-value pair > is part of a compaction … If you take the total number of bytes written to > disk … and divide that by the number of bytes you would have to write if you > simply wrote an append-only log … you get the write amplification. The book defines the metric and names the three sources. Here you measure it on a real engine. ## Prerequisites The surprise is about the LSM write path. These help; each line notes where it shows up. 1. **LSM-tree write path** — writes go to an in-memory *memtable* (logged to the WAL for durability); when it fills, it is flushed to an immutable, sorted *SSTable* on disk. 2. **Compaction and levels** — to bound reads and reclaim space, background compaction merges SSTables and moves data down through levels (L0 → L1 → …). Each compaction *rewrites* the data it touches. 3. **Write amplification** — total bytes written to storage ÷ bytes of logical data (what a plain append-only log would write). The number you compute. 4. **Incompressible data** — random values, so the on-disk SSTable size tracks the logical data and the rewrites aren't hidden by compression. ### Where to learn them - **LSM write path & compaction (#1–#2):** DDIA §"Log-Structured Storage"; the RocksDB wiki pages "Leveled Compaction" and "Compaction." - **Write vs space vs read amplification (#3):** the **RUM conjecture** (Athanassoulis et al.) — you trade off Read, Update (write), and Memory/space; you can't minimize all three. - **Reducing it for big values:** the **WiscKey** paper (key-value separation), which the book alludes to. If only one is new, make it #2 — compaction rewriting data per level is where the amplification comes from. ## Environment The exact factor varies a little between runs — background compaction is timing-dependent — but the shape holds (a single-digit multiple, dominated by compaction): - macOS 26.5.2 (Darwin 25.5.0), arm64 - `rocksdict` 0.3.29 (bundling RocksDB) on Python 3.12.13, run via `uv` — no Docker - 2,000,000 records, ~230 MB of key+value data, incompressible random values, a 4 MB memtable so data cascades through several levels ## Mental model We insert 2,000,000 records — a 12-byte key and a 100-byte random value each, ~230 MB of logical data — into RocksDB configured with a small (4 MB) memtable and small level sizes, so the data has to cascade down through several levels of compaction. Then we read RocksDB's own byte counters and add up everything it wrote to disk: - **WAL** — every record, logged once for durability. - **Flush** — each memtable written out to an L0 SSTable. - **Compaction** — the data rewritten each time it is merged down a level. Write amplification is that total over the 230 MB of logical data. The whole exercise is one script, `code/write_amplification.py`. ## Setup ``` cd study/ddia/ch04/code # RocksDB via the rocksdict package, installed into a throwaway uv env. No Docker. uv run --with rocksdict python3 write_amplification.py # ~60 s ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — the append-only baseline (predict) **Predict.** You have ~230 MB of key/value data to store durably. A plain append-only log — the simplest durable store — writes each record once as it arrives. How many megabytes does *that* write to disk? (About 230 MB: the data, once, sequentially.) Hold that as the baseline. ### Step 2 — what the LSM actually writes (the surprise) **Predict.** Now the same 230 MB into an LSM-tree, which the book praises for *sequential* writes and high write throughput. How many megabytes does it write to disk to store it? (The instinct: "a bit more than the log — the WAL plus the SSTables, so maybe 2×.") **Do.** Run the script. **Observe.** ``` inserted 2,000,000 records (~230 MB of key+value data) into RocksDB (an LSM engine) what RocksDB actually wrote to disk to store 230 MB durably: write-ahead log (WAL, written once) = 260 MB memtable flushes to L0 = 238 MB compaction (rewriting SSTables) = 1593 MB ---------------------------------------------------- total written to disk = 2091 MB write amplification = 2091 / 230 = 9.1x (vs a plain append-only log) compaction alone rewrote the data 6.9x as it migrated down the levels ``` **2,091 MB to store 230 MB — a 9.1× write amplification.** The WAL and the initial flush are each ~1× the data, as expected. The surprise is **compaction: 1,593 MB**, rewriting the same data ~7× as it was merged down level after level. ### Step 3 — it isn't wasted space **Predict.** After writing 2 GB, how big is the database on disk? **Observe.** ``` final database size on disk = 235 MB (~= the 230 MB of data; the rewrites are transient work, not waste) ``` The database is **235 MB** — essentially just the data. The 2 GB wasn't stored; it was *written and overwritten*. Write amplification is wasted **write work and disk endurance**, not wasted space — the rewrites churned through the SSD to keep the data sorted and compact. ## What you should see - Append-only baseline: ~**230 MB** written (the data, once). - LSM total to disk: ~**2,091 MB** — WAL ~260, flush ~238, **compaction ~1,593**. - **Write amplification ≈ 9×**, compaction ~7× of it. - Final on-disk size ≈ **235 MB** — the amplification is write work, not storage. ## Why An LSM-tree never overwrites data in place; it only appends and later merges. Each record is written to the WAL the moment it arrives (write #1), so a crash can't lose it. It sits in the in-memory memtable until that fills, then the whole memtable is flushed to an immutable L0 SSTable (write #2). From there, background compaction repeatedly merges SSTables and pushes the data down through the levels — L0 → L1 → … → L6 — to keep the number of files a read must check bounded and to discard superseded versions. Every one of those merges *rewrites* the data it touches, so a record is written once more for roughly each level it descends through (writes #3, #4, …). Add it up and storing 230 MB cost 2,091 MB of writes: the WAL and flush are ~1× each, and compaction is the ~7× that dominates. That sounds like a pure loss until you remember the flip side from the random-vs-sequential exercise. *Every* one of those 2 GB of writes is **sequential** — whole SSTables written start to finish — because the LSM never seeks to update a page in place. A B-tree has far *lower* write amplification (each change writes the WAL plus the page, ≥2×, occasionally a full page for crash safety) but those writes are **random**, scattered across the tree. So the two designs sit at opposite corners: the LSM writes many more bytes but does so sequentially, and can sustain higher write throughput on the same disk; the B-tree writes fewer bytes but pays the random-write tax. That is the "it depends" the book leaves you with — and why write-heavy, random-key workloads lean LSM. **The boundary — write amplification is a knob, traded against space and reads.** The 9× here is a consequence of *leveled* compaction, which minimizes wasted space and read amplification at the cost of write amplification. Choose *size-tiered* / *universal* compaction and write amplification drops sharply, but space amplification rises (stale copies linger). Enlarge the memtable or reduce the number of levels and the factor falls too. You cannot minimize write, space, and read amplification at once — that is the RUM conjecture — so every LSM ships a compaction strategy that picks a point in that space. (And when values are large, key-value separation — WiscKey / BlobDB — cuts write amplification by compacting only the keys and leaving the values put.) ### Go deeper 1. **Turn compression back on.** Remove the `set_compression_type(none)` line and re-run. Predict what happens to the byte counts and to the *ratio* — and why the ratio is a fair comparison only when logical and physical bytes are measured the same way. 2. **Grow the data.** Double `N` to 4,000,000. Predict whether write amplification goes up (more data → one more level to cascade through) and by roughly how much. 3. **Bigger memtable.** Raise `set_write_buffer_size` to 64 MB. Predict how much the amplification drops when far more writes are absorbed in memory before each flush.