study › DDIA › Chapter 5 · Encoding and Evolution

The Same Record, 81 Bytes to 32

One small record encoded five ways falls in a clean ladder — JSON 81 → MessagePack 66 → Protobuf 33 → Avro 32 bytes. Each rung drops a layer of self-description, and 38% of the JSON turns out to be field names, re-sent in every record.

Concept

Encode one small record five ways and its size falls in a clean ladder: JSON spells out every field name and value as text; MessagePack is binary but, having no schema, still carries every field name; Protocol Buffers replaces the names with 1-byte field tags because the schema holds the names; Avro drops even the tags, storing bare values in schema order. You will predict the sizes and watch 81 → 66 → 33 → 32 bytes — each rung dropping a layer of self-description, with 38% of the JSON turning out to be field names repeated in every record.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 5 — Encoding and Evolution, §"Binary encodings" (and §"Protocol Buffers", §"Avro"):

since they don't prescribe a schema, they need to include all the object field names within the encoded data.

The book walks the same record down from JSON through Protobuf to Avro. Here you encode it and measure each rung — and see exactly which bytes each format spends.

Prerequisites

The surprise is about what each encoding must carry besides the values. These help; each line notes where it shows up.

Self-describing vs. schema-based encodings — JSON/MessagePack embed the field names so the bytes describe themselves; Protobuf/Avro rely on a schema both sides share, so the bytes carry only data (plus tags, for Protobuf).
Field tags — Protobuf identifies each field by a small integer tag, not its name; the tag costs ~1 byte where a name costs its length in characters.
Schema order — Avro writes fields in the order the schema declares, with no tags at all, so decoding requires that exact schema (see the Avro exercise).
The record model — a small object with a string, an integer, and a list; the same across all five encodings so only the encoding varies.

Where to learn the prerequisites The encodings themselves (#1–#3): DDIA §"Binary encodings", §"Protocol Buffers", §"Avro"; the Protobuf and Avro "encoding" spec pages. Why field names dominate small records (#2): the MessagePack spec (it is "JSON, but binary" — same data model, so same field-name overhead). If only one is new, make it #1 — schema-based vs self-describing is the whole ladder.

Environment these numbers came from Deterministic, so the byte counts reproduce exactly. Captured on macOS 26.5.2 (Darwin 25.5.0), arm64, Python 3.12.13 via uv, with msgpack, fastavro, and Protocol Buffers (grpcio-tools compiling the .proto at runtime + the protobuf runtime) — no Docker. One record: {userName: "Martin", favoriteNumber: 1337, interests: ["daydreaming", "hacking"]}.

Mental model

The same record, encoded five ways:

JSON — text; every field name, value, and bit of punctuation is spelled out.
MessagePack — a binary JSON: compact type/length prefixes instead of quotes and brackets, but with no schema it still stores every field name.
Protocol Buffers — a schema assigns each field a tag number; the encoded bytes carry (tag, value) pairs, so a 1-byte tag stands in for the field name.
Avro — a schema fixes the field order; the bytes are just the values, concatenated, with no names and no tags.

We encode the record in each and compare len(bytes). The whole exercise is one script, code/encoding_size_ladder.py.

Setup

cd study/ddia/ch05/code
uv run --with msgpack --with fastavro --with grpcio-tools --with protobuf \
    python3 encoding_size_ladder.py

Do not read the output yet — make each prediction first.

Step 1 · predict the ladder

Predict. The record has three fields. Write down your guess for the encoded size in JSON, MessagePack, Protobuf, and Avro — in bytes. (Most people guess JSON around 80, and then that the binary formats are "maybe half.")

Step 2 · measure it

Do. Run the script.

How do the four encodings' sizes compare?

encoded size, and what each format spends its bytes on: JSON 81 bytes text: every field name + value + punctuation MessagePack 66 bytes binary, but schemaless -> still carries every field name Protocol Buffers 33 bytes field names become 1-byte tags (the schema holds the names) Avro 32 bytes bare values in schema order -- no names, no tags

81 → 66 → 33 → 32. MessagePack shaves only ~18% off JSON — because its bytes are still mostly field names, just without the quotes and colons. The big drop is JSON/MessagePack → Protobuf (2×), where a schema lets a 1-byte tag replace each spelled-out name. Avro squeezes out the last byte by dropping the tags too.

Step 3 · where JSON's bytes went

Predict. Of JSON's 81 bytes, how many are the field names — the characters userName, favoriteNumber, interests?

How much of the JSON is just field names?

of JSON's 81 bytes, 31 are field-name characters (38%) -- repeated in every record. the ladder: 81 -> 66 -> 33 -> 32 bytes, each rung dropping a layer of self-description.

38% of the JSON is field names — userName, favoriteNumber, interests — and those characters are re-sent in every single record. That is the overhead a schema removes: send the names once (in the schema), not a billion times (in the data).

What you should see

The ladder: JSON 81 → MessagePack 66 → Protobuf 33 → Avro 32 bytes.
MessagePack is only ~18% smaller than JSON; the 2× drop is at the schema boundary (→ Protobuf).
~38% of the JSON encoding is field-name characters, repeated per record.

Why

Every encoding must store the values; the ladder is about how much else it stores. JSON is self-describing text: to know that 1337 is the favorite number, the bytes literally contain the string "favoriteNumber" next to it, plus quotes, a colon, and commas. MessagePack keeps JSON's data model but swaps the textual syntax for compact binary type/length markers — so it loses the punctuation but, lacking any schema, must still embed every field name. That is why it only trims ~18%: on a small record, the names are most of the bytes.

Protocol Buffers and Avro break that by moving the field names out of the data and into a schema both sides already have. Protobuf tags each field with a small integer, so the wire format is a sequence of (tag, value) pairs and a 1-byte tag replaces an 8-to-14-character name — halving the size. Avro goes further: because the schema also fixes the field order, Avro doesn't even need tags; it writes the values back to back and nothing else, reaching the floor of "just the data." The price of that last byte is steep in a different currency — Avro's bytes are meaningless without the exact writing schema — which the Avro exercise makes concrete.

The boundary — the ladder is steepest for small records, and schemas cost elsewhere Field-name overhead is a fixed cost per record, so it dominates small messages (here, 38% of JSON) and fades on large ones with big values or long arrays — a record that is mostly a 10 KB text blob encodes to nearly the same size in all four formats. And "smaller on the wire" is not free: JSON needs no schema and is human-readable and debuggable; Protobuf and Avro demand that both sides agree on a schema and add a build/codegen step. The next exercise (compression) shows the raw size gap can also shrink once you gzip — so the size ladder is the start of the trade-off, not the whole of it.

Go deeper

Grow the values. Replace the short strings with a 5 KB text field and re-run. Predict what happens to the ratio between JSON and Avro as the fixed field-name overhead is dwarfed by the payload.
A batch of records. Encode 100,000 copies. Predict how the per-record overhead (JSON re-sends field names each time; Avro's object-container file writes the schema once) changes the totals.
Add a field. Extend the record with a fourth field and re-measure each format's growth — one value + one name (JSON), one value + one tag (Protobuf), one value (Avro). This previews schema evolution.

Sources: DDIA 2e, Ch. 5, §"Binary encodings", §"Protocol Buffers", §"Avro" · the MessagePack, Protobuf, and Avro encoding specs.