# DDIA Ch. 5: the 2.74× binary win that gzip shrinks to 1.43× ## Concept Encode a batch of 10,000 records four ways and the binary formats look like a runaway win: Avro is **2.74× smaller than JSON** on the raw bytes. But most of JSON's bulk is the *same* field names and punctuation repeated 10,000 times — which is precisely the redundancy gzip was built to crush. So gzip each blob and watch the advantage largely collapse: after compression Avro is only **1.43× smaller** than JSON, because gzip already removed the very overhead that made the binary formats look good. You will predict the raw gap, then predict what survives compression — and find the size argument for binary mostly evaporates. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 5 — *Encoding and Evolution*, §"Binary encodings" (and §"The Merits of Schemas"): > a compact encoding is not the only reason to use these formats. The book lists schema formats' merits beyond size. This exercise pressure-tests the size merit itself: measure the raw gap, then compress and see how little of it is left. ## Prerequisites The surprise is that a "2-3× smaller" number is mostly compressible overhead. These help; each line notes where it shows up. 1. **Self-describing vs. schema-based encodings** — JSON/MessagePack embed the field names in every record, so a 10,000-record batch re-sends `"userName"` 10,000 times; Protobuf/Avro keep the names in a schema instead. This is the redundancy that inflates the raw JSON and that gzip later removes. 2. **How gzip (DEFLATE) works** — it replaces repeated byte sequences with short back-references (LZ77) plus entropy coding. Repeated field names are ideal prey, so the format with the *most* repetition compresses the *most*. 3. **Raw size vs. compressed size** — "smaller on the wire" depends on whether the wire is compressed. The whole point is that these two rankings differ: the raw table and the gzipped table tell different stories. 4. **A batch, not one record** — compression needs volume to find repetition, so we encode 10,000 records; on a single tiny message gzip has almost nothing to work with (and its header would cost more than it saves). ### Where to learn them - **The encodings (#1):** DDIA §"Binary encodings", §"Protocol Buffers", §"Avro"; and the size-ladder exercise, which measures the per-record overhead directly. - **DEFLATE / gzip (#2):** RFC 1951 (DEFLATE) and RFC 1952 (gzip); any LZ77 explainer. You only need the intuition "repeats get cheap." - **Why batch size matters (#3–#4):** try the "Go deeper" variant that shrinks the batch to 1 record and watch the ranking flip back. If only one is new, make it #2 — that gzip preys on exactly JSON's repeated field names is the hinge of the whole result. ## Environment Deterministic, so the byte counts reproduce exactly: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.12.13 via `uv`, with `msgpack`, `fastavro`, and Protocol Buffers (`grpcio-tools` compiling the `.proto` at runtime + the `protobuf` runtime) — no Docker; gzip is stdlib - A batch of 10,000 user records `{id, userName: "user{i}", favoriteNumber, interests[2]}`, generated from a fixed formula (no randomness), gzipped at level 9 ## Mental model The same 10,000-record batch, measured twice: - **Raw** — encode the whole batch in each format and count the bytes. JSON (newline-delimited) spells out every field name in every record; MessagePack is binary but still carries the names; Protobuf uses 1-byte tags; Avro writes bare values in schema order. Raw sizes fall in the familiar ladder. - **Gzipped** — run each blob through `gzip.compress(blob, 9)` and count again. gzip finds and collapses repeated byte sequences — and JSON's biggest feature is repetition. So the format that looked *worst* raw has the *most* to gain from compression, and the four sizes bunch back together. We compare each format to JSON both raw and gzipped. The whole exercise is one script, `code/compression_gap.py`. ## Setup ``` cd study/ddia/ch05/code uv run --with msgpack --with fastavro --with grpcio-tools --with protobuf \ python3 compression_gap.py ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — predict the raw gap **Predict.** Encode 10,000 records. JSON as newline-delimited objects, and Avro as concatenated schemaless values. How much smaller is the raw Avro blob than the raw JSON blob — 1.5×? 2×? 3×? Write down a number. ### Step 2 — measure raw, then gzipped **Do.** Run the script. **Observe.** ``` batch of 10,000 user records {id, userName, favoriteNumber, interests[2]} format raw vs JSON gzip -9 vs JSON -------------------------------------------------------------- JSON (ndjson) 924,160 1.00x 97,092 1.00x MessagePack 722,855 0.78x 89,076 0.92x Protocol Buffers 374,603 0.41x 75,805 0.78x Avro 337,305 0.36x 68,076 0.70x ``` **Raw, the binary formats win big:** Avro is `0.36×` JSON's size — **2.74× smaller**. That is the number people quote when they say "switch to Protobuf and cut your bytes in half." Look at the raw column and binary looks like an easy win. ### Step 3 — predict what survives compression, then read the gzip column **Predict.** Now cover the `gzip` columns. gzip compresses all four blobs. Does the 2.74× Avro-vs-JSON gap *hold* after compression, *widen*, or *shrink*? **Observe.** ``` raw: Avro is 2.74x smaller than JSON (0.36x its size). gzip: Avro is 1.43x smaller than JSON (0.70x its size) -- the gap largely closed. gzip shrinks JSON by 89% (924,160 -> 97,092 bytes): most of JSON's bulk was repeated field names + punctuation, which is exactly what gzip removes. ``` **The gap collapses from 2.74× to 1.43×.** gzip crushes the 924 KB JSON blob by **89%**, down to 97 KB, because those 10,000 repeats of `"userName"`, `"favoriteNumber"`, `"interests"`, and the braces and quotes are exactly the redundancy DEFLATE eats. Avro started with far less to remove (337 KB → 68 KB), so the two end up close. The raw 2.74× advantage was mostly *compressible overhead* — it never survived the wire once you turned compression on. ## What you should see - Raw: **JSON 924,160 → MessagePack 722,855 → Protobuf 374,603 → Avro 337,305** bytes; Avro is **2.74×** smaller than JSON. - Gzipped: **JSON 97,092 → MessagePack 89,076 → Protobuf 75,805 → Avro 68,076** bytes; Avro is only **1.43×** smaller than JSON. - gzip shrinks the JSON blob by **89%** — most of it was repeated field names and punctuation, exactly what DEFLATE removes. ## Why Every format stores the same values; they differ in how much *self-description* they add. JSON adds the most: in a 10,000-record batch the string `"userName"` appears 10,000 times, `"favoriteNumber"` 10,000 times, plus braces, quotes, colons, and commas throughout. That is why the raw JSON blob is 924 KB while Avro — which carries the field names *once*, in a schema the reader already has — is 337 KB. On the raw bytes, dropping repeated self-description is worth 2.74×. But "repeated byte sequences" is the exact thing gzip's LZ77 stage is designed to eliminate: the first `"favoriteNumber":` costs its full length; every later copy becomes a short back-reference pointing at the first. So the format with the most repetition has the most to gain from compression. gzip takes 89% off JSON but far less off Avro, whose bytes were already close to pure payload with little for the compressor to find. The rankings that looked so different raw converge: Avro's 2.74× lead shrinks to 1.43×. Compression, in effect, gives JSON a schema of its own — a dynamically built dictionary of the repeated strings — reclaiming most of what a static schema bought Protobuf and Avro. **The boundary — binary's size win is real only where you *can't* compress a batch.** The 2.74× is an artifact of measuring uncompressed bytes; put a gzip (or brotli, or zstd) between the encoder and the wire and it falls to ~1.4×. That compression needs *volume* to find repetition — so binary's size edge genuinely holds for the cases gzip can't help: a single tiny message (gzip's header would cost more than it saves), a latency- or CPU-bound path where you won't pay to compress, or per-message framing on a hot queue. Outside those, choose Protobuf or Avro for their *other* merits — a checked schema, fast typed parsing, safe evolution, code generation — not for a size win that compression already gave you for free. That is exactly the book's point that "a compact encoding is not the only reason to use these formats." ### Go deeper 1. **Shrink the batch to 1 record.** Set `N = 1` and re-run. Predict what gzip does to a ~90-byte JSON object (hint: gzip has a fixed header and needs repetition to win) — and watch the raw ranking survive because there is nothing to compress. 2. **Swap the compressor.** Try `zstandard` at a high level, or `brotli`, in place of gzip. Predict whether a stronger compressor closes the JSON↔Avro gap *more* (better at the repeats) or reveals a residual binary edge. 3. **Make the values incompressible.** Replace `userName: "user{i}"` with a random 40-char token per record so the field *values*, not just the names, dominate. Predict how the raw ratio and the gzipped ratio both move as the payload stops being repetitive.