An LSM-tree logs each value, flushes it to an SSTable, and rewrites it on every compaction as it sinks down the levels. Store ~230 MB in RocksDB and it writes 2,091 MB to disk — a 9× write amplification, ~7× of it compaction — while the database ends up just 235 MB. The write work is the price of writing only sequentially.
One logical write becomes many physical writes. An LSM-tree logs each value to its write-ahead log, writes it again when the memtable is flushed to an SSTable, and writes it again every time compaction rewrites the SSTable it lives in as the data migrates down the levels. The book calls the ratio — total bytes written to disk over the bytes a plain append-only log would write — write amplification. You will store ~230 MB of data in a real LSM engine (RocksDB) and watch it write 2,091 MB to disk to do it: a 9× amplification, most of it compaction rewriting the same data over and over.
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 4 — Storage and Retrieval, §"Write amplification":
With LSM-trees, a value is first written to the log for durability, then again when the memtable is written to disk, and again every time the key-value pair is part of a compaction … If you take the total number of bytes written to disk … and divide that by the number of bytes you would have to write if you simply wrote an append-only log … you get the write amplification.
The book defines the metric and names the three sources. Here you measure it on a real engine.
The surprise is about the LSM write path. These help; each line notes where it shows up.
rocksdict 0.3.29 (bundling RocksDB) on Python 3.12.13, run via uv — no Docker. 2,000,000 records, ~230 MB of key+value data, incompressible random values, a 4 MB memtable so data cascades through several levels.
We insert 2,000,000 records — a 12-byte key and a 100-byte random value each, ~230 MB of logical data — into RocksDB configured with a small (4 MB) memtable and small level sizes, so the data has to cascade down through several levels of compaction. Then we read RocksDB's own byte counters and add up everything it wrote to disk:
Write amplification is that total over the 230 MB of logical data. The whole exercise is one script, code/write_amplification.py.
cd study/ddia/ch04/code
# RocksDB via the rocksdict package, installed into a throwaway uv env. No Docker.
uv run --with rocksdict python3 write_amplification.py # ~60 sDo not read the output yet — make each prediction first.
Predict. You have ~230 MB of key/value data to store durably. A plain append-only log — the simplest durable store — writes each record once as it arrives. How many megabytes does that write to disk? (About 230 MB: the data, once, sequentially.) Hold that as the baseline.
Predict. Now the same 230 MB into an LSM-tree, which the book praises for sequential writes and high write throughput. How many megabytes does it write to disk to store it? (The instinct: "a bit more than the log — the WAL plus the SSTables, so maybe 2×.")
Do. Run the script.
2,091 MB to store 230 MB — a 9.1× write amplification. The WAL and the initial flush are each ~1× the data, as expected. The surprise is compaction: 1,593 MB, rewriting the same data ~7× as it was merged down level after level.
Predict. After writing 2 GB, how big is the database on disk?
The database is 235 MB — essentially just the data. The 2 GB wasn't stored; it was written and overwritten. Write amplification is wasted write work and disk endurance, not wasted space — the rewrites churned through the SSD to keep the data sorted and compact.
An LSM-tree never overwrites data in place; it only appends and later merges. Each record is written to the WAL the moment it arrives (write #1), so a crash can't lose it. It sits in the in-memory memtable until that fills, then the whole memtable is flushed to an immutable L0 SSTable (write #2). From there, background compaction repeatedly merges SSTables and pushes the data down through the levels — L0 → L1 → … → L6 — to keep the number of files a read must check bounded and to discard superseded versions. Every one of those merges rewrites the data it touches, so a record is written once more for roughly each level it descends through (writes #3, #4, …). Add it up and storing 230 MB cost 2,091 MB of writes: the WAL and flush are ~1× each, and compaction is the ~7× that dominates.
That sounds like a pure loss until you remember the flip side from the random-vs-sequential exercise. Every one of those 2 GB of writes is sequential — whole SSTables written start to finish — because the LSM never seeks to update a page in place. A B-tree has far lower write amplification (each change writes the WAL plus the page, ≥2×, occasionally a full page for crash safety) but those writes are random, scattered across the tree. So the two designs sit at opposite corners: the LSM writes many more bytes but does so sequentially, and can sustain higher write throughput on the same disk; the B-tree writes fewer bytes but pays the random-write tax. That is the "it depends" the book leaves you with — and why write-heavy, random-key workloads lean LSM.
set_compression_type(none) line and re-run. Predict what happens to the byte counts and to the ratio — and why the ratio is a fair comparison only when logical and physical bytes are measured the same way.N to 4,000,000. Predict whether write amplification goes up (more data → one more level to cascade through) and by roughly how much.set_write_buffer_size to 64 MB. Predict how much the amplification drops when far more writes are absorbed in memory before each flush.Sources: DDIA 2e, Ch. 4, §"Write amplification" · RocksDB wiki, "Leveled Compaction" · Athanassoulis et al., "Designing Access Methods: The RUM Conjecture" · Lu et al., "WiscKey."