Compression Closes the Gap

On raw bytes, Avro is 2.74× smaller than JSON — the number people quote to justify switching. But JSON's bulk is repeated field names, exactly what gzip removes. Compress the batch and the advantage collapses to 1.43×: the size win was mostly compressible overhead.

Concept

Encode a batch of 10,000 records four ways and the binary formats look like a runaway win: Avro is 2.74× smaller than JSON on the raw bytes. But most of JSON's bulk is the same field names and punctuation repeated 10,000 times — which is precisely the redundancy gzip was built to crush. So gzip each blob and watch the advantage largely collapse: after compression Avro is only 1.43× smaller than JSON, because gzip already removed the very overhead that made the binary formats look good. You will predict the raw gap, then predict what survives compression — and find the size argument for binary mostly evaporates.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 5 — Encoding and Evolution, §"Binary encodings" (and §"The Merits of Schemas"):

a compact encoding is not the only reason to use these formats.

The book lists schema formats' merits beyond size. This exercise pressure-tests the size merit itself: measure the raw gap, then compress and see how little of it is left.

Prerequisites

The surprise is that a "2-3× smaller" number is mostly compressible overhead. These help; each line notes where it shows up.

1. Self-describing vs. schema-based encodings — JSON/MessagePack embed the field names in every record, so a 10,000-record batch re-sends "userName" 10,000 times; Protobuf/Avro keep the names in a schema instead. This is the redundancy that inflates the raw JSON and that gzip later removes.
2. How gzip (DEFLATE) works — it replaces repeated byte sequences with short back-references (LZ77) plus entropy coding. Repeated field names are ideal prey, so the format with the most repetition compresses the most.
3. Raw size vs. compressed size — "smaller on the wire" depends on whether the wire is compressed. The whole point is that these two rankings differ: the raw table and the gzipped table tell different stories.
4. A batch, not one record — compression needs volume to find repetition, so we encode 10,000 records; on a single tiny message gzip has almost nothing to work with (and its header would cost more than it saves).

Mental model

The same 10,000-record batch, measured twice:

• Raw — encode the whole batch in each format and count the bytes. JSON (newline-delimited) spells out every field name in every record; MessagePack is binary but still carries the names; Protobuf uses 1-byte tags; Avro writes bare values in schema order. Raw sizes fall in the familiar ladder.
• Gzipped — run each blob through gzip.compress(blob, 9) and count again. gzip finds and collapses repeated byte sequences — and JSON's biggest feature is repetition. So the format that looked worst raw has the most to gain from compression, and the four sizes bunch back together.

We compare each format to JSON both raw and gzipped. The whole exercise is one script, code/compression_gap.py.

Setup

cd study/ddia/ch05/code
uv run --with msgpack --with fastavro --with grpcio-tools --with protobuf \
    python3 compression_gap.py

Do not read the output yet — make each prediction first.

What you should see

• Raw: JSON 924,160 → MessagePack 722,855 → Protobuf 374,603 → Avro 337,305 bytes; Avro is 2.74× smaller than JSON.
• Gzipped: JSON 97,092 → MessagePack 89,076 → Protobuf 75,805 → Avro 68,076 bytes; Avro is only 1.43× smaller than JSON.
• gzip shrinks the JSON blob by 89% — most of it was repeated field names and punctuation, exactly what DEFLATE removes.

Why

Every format stores the same values; they differ in how much self-description they add. JSON adds the most: in a 10,000-record batch the string "userName" appears 10,000 times, "favoriteNumber" 10,000 times, plus braces, quotes, colons, and commas throughout. That is why the raw JSON blob is 924 KB while Avro — which carries the field names once, in a schema the reader already has — is 337 KB. On the raw bytes, dropping repeated self-description is worth 2.74×.

But "repeated byte sequences" is the exact thing gzip's LZ77 stage is designed to eliminate: the first "favoriteNumber": costs its full length; every later copy becomes a short back-reference pointing at the first. So the format with the most repetition has the most to gain from compression. gzip takes 89% off JSON but far less off Avro, whose bytes were already close to pure payload with little for the compressor to find. The rankings that looked so different raw converge: Avro's 2.74× lead shrinks to 1.43×. Compression, in effect, gives JSON a schema of its own — a dynamically built dictionary of the repeated strings — reclaiming most of what a static schema bought Protobuf and Avro.

Go deeper

• Shrink the batch to 1 record. Set N = 1 and re-run. Predict what gzip does to a ~90-byte JSON object (hint: gzip has a fixed header and needs repetition to win) — and watch the raw ranking survive because there is nothing to compress.
• Swap the compressor. Try zstandard at a high level, or brotli, in place of gzip. Predict whether a stronger compressor closes the JSON↔Avro gap more (better at the repeats) or reveals a residual binary edge.
• Make the values incompressible. Replace userName: "user{i}" with a random 40-char token per record so the field values, not just the names, dominate. Predict how the raw ratio and the gzipped ratio both move as the payload stops being repetitive.

Sources: DDIA 2e, Ch. 5, §"Binary encodings", §"The Merits of Schemas" · RFC 1951 (DEFLATE), RFC 1952 (gzip) · the MessagePack, Protobuf, and Avro encoding specs.