# DDIA Ch. 5: the same record, 81 bytes to 32

## Concept

Encode one small record five ways and its size falls in a clean ladder: JSON
spells out every field name and value as text; MessagePack is binary but, having
no schema, still carries every field *name*; Protocol Buffers replaces the names
with 1-byte field tags because the schema holds the names; Avro drops even the
tags, storing bare values in schema order. You will predict the sizes and watch
**81 → 66 → 33 → 32 bytes** — each rung dropping a layer of self-description,
with 38% of the JSON turning out to be field names repeated in every record.

## Provenance

*Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini,
2025), Chapter 5 — *Encoding and Evolution*, §"Binary encodings" (and §"Protocol
Buffers", §"Avro"):

> since they don't prescribe a schema, they need to include all the object field
> names within the encoded data.

The book walks the same record down from JSON through Protobuf to Avro. Here you
encode it and measure each rung — and see exactly which bytes each format spends.

## Prerequisites

The surprise is about what each encoding must carry besides the values. These
help; each line notes where it shows up.

1. **Self-describing vs. schema-based encodings** — JSON/MessagePack embed the
   field names so the bytes describe themselves; Protobuf/Avro rely on a schema
   both sides share, so the bytes carry only data (plus tags, for Protobuf).
2. **Field tags** — Protobuf identifies each field by a small integer tag, not
   its name; the tag costs ~1 byte where a name costs its length in characters.
3. **Schema order** — Avro writes fields in the order the schema declares, with
   no tags at all, so decoding requires that exact schema (see the Avro exercise).
4. **The record model** — a small object with a string, an integer, and a list;
   the same across all five encodings so only the *encoding* varies.

### Where to learn them

- **The encodings themselves (#1–#3):** DDIA §"Binary encodings", §"Protocol
  Buffers", §"Avro"; the Protobuf and Avro "encoding" spec pages.
- **Why field names dominate small records (#2):** the MessagePack spec (it is
  "JSON, but binary" — same data model, so same field-name overhead).

If only one is new, make it #1 — schema-based vs self-describing is the whole
ladder.

## Environment

Deterministic, so the byte counts reproduce exactly:

- macOS 26.5.2 (Darwin 25.5.0), arm64
- Python 3.12.13 via `uv`, with `msgpack`, `fastavro`, and Protocol Buffers
  (`grpcio-tools` compiling the `.proto` at runtime + the `protobuf` runtime) —
  no Docker
- One record: `{userName: "Martin", favoriteNumber: 1337, interests:
  ["daydreaming", "hacking"]}`

## Mental model

The same record, encoded five ways:

- **JSON** — text; every field name, value, and bit of punctuation is spelled out.
- **MessagePack** — a binary JSON: compact type/length prefixes instead of quotes
  and brackets, but with no schema it still stores every field *name*.
- **Protocol Buffers** — a schema assigns each field a tag number; the encoded
  bytes carry `(tag, value)` pairs, so a 1-byte tag stands in for the field name.
- **Avro** — a schema fixes the field order; the bytes are just the values,
  concatenated, with no names and no tags.

We encode the record in each and compare `len(bytes)`. The whole exercise is one
script, `code/encoding_size_ladder.py`.

## Setup

```
cd study/ddia/ch05/code
uv run --with msgpack --with fastavro --with grpcio-tools --with protobuf \
    python3 encoding_size_ladder.py
```

Do **not** read the output yet — make each prediction first.

## Steps

### Step 1 — predict the ladder

**Predict.** The record has three fields. Write down your guess for the encoded
size in **JSON**, **MessagePack**, **Protobuf**, and **Avro** — in bytes. (Most
people guess JSON around 80, and then that the binary formats are "maybe half.")

### Step 2 — measure it

**Do.** Run the script.

**Observe.**

```
encoded size, and what each format spends its bytes on:
    JSON               81 bytes   text: every field name + value + punctuation
    MessagePack        66 bytes   binary, but schemaless -> still carries every field name
    Protocol Buffers   33 bytes   field names become 1-byte tags (the schema holds the names)
    Avro               32 bytes   bare values in schema order -- no names, no tags
```

**81 → 66 → 33 → 32.** MessagePack shaves only ~18% off JSON — because its bytes
are *still mostly field names*, just without the quotes and colons. The big drop
is JSON/MessagePack → Protobuf (2×), where a schema lets a 1-byte tag replace
each spelled-out name. Avro squeezes out the last byte by dropping the tags too.

### Step 3 — where JSON's bytes went

**Observe.**

```
of JSON's 81 bytes, 31 are field-name characters (38%) -- repeated in every record.
the ladder: 81 -> 66 -> 33 -> 32 bytes, each rung dropping a layer of self-description.
```

**38% of the JSON is field names** — `userName`, `favoriteNumber`, `interests` —
and those characters are re-sent in *every single record*. That is the overhead a
schema removes: send the names once (in the schema), not a billion times (in the
data).

## What you should see

- The ladder: **JSON 81 → MessagePack 66 → Protobuf 33 → Avro 32** bytes.
- MessagePack is only ~18% smaller than JSON; the 2× drop is at the
  schema boundary (→ Protobuf).
- **~38%** of the JSON encoding is field-name characters, repeated per record.

## Why

Every encoding must store the *values*; the ladder is about how much *else* it
stores. JSON is self-describing text: to know that `1337` is the favorite number,
the bytes literally contain the string `"favoriteNumber"` next to it, plus quotes,
a colon, and commas. MessagePack keeps JSON's data model but swaps the textual
syntax for compact binary type/length markers — so it loses the punctuation but,
lacking any schema, must still embed every field name. That is why it only trims
~18%: on a small record, the names *are* most of the bytes.

Protocol Buffers and Avro break that by moving the field names out of the data
and into a schema both sides already have. Protobuf tags each field with a small
integer, so the wire format is a sequence of `(tag, value)` pairs and a 1-byte
tag replaces an 8-to-14-character name — halving the size. Avro goes further:
because the schema also fixes the field *order*, Avro doesn't even need tags; it
writes the values back to back and nothing else, reaching the floor of "just the
data." The price of that last byte is steep in a different currency — Avro's
bytes are meaningless without the exact writing schema — which the Avro exercise
makes concrete.

**The boundary — the ladder is steepest for small records, and schemas cost
elsewhere.** Field-name overhead is a *fixed* cost per record, so it dominates
small messages (here, 38% of JSON) and fades on large ones with big values or
long arrays — a record that is mostly a 10 KB text blob encodes to nearly the
same size in all four formats. And "smaller on the wire" is not free: JSON needs
no schema and is human-readable and debuggable; Protobuf and Avro demand that
both sides agree on a schema and add a build/codegen step. The next exercise
(compression) shows the raw size gap can also shrink once you gzip — so the size
ladder is the *start* of the trade-off, not the whole of it.

### Go deeper

1. **Grow the values.** Replace the short strings with a 5 KB text field and
   re-run. Predict what happens to the *ratio* between JSON and Avro as the
   fixed field-name overhead is dwarfed by the payload.
2. **A batch of records.** Encode 100,000 copies. Predict how the per-record
   overhead (JSON re-sends field names each time; Avro's object-container file
   writes the schema once) changes the totals.
3. **Add a field.** Extend the record with a fourth field and re-measure each
   format's growth — one value + one name (JSON), one value + one tag (Protobuf),
   one value (Avro). This previews schema evolution.