# DDIA Ch. 5: Avro is the smallest — and unreadable without its schema

## Concept

Avro strips a record down to bare values in schema order: no field names, no tags,
not even a boundary between one field and the next. That makes it the most compact
of the binary encodings — here **21 bytes to Protobuf's 24** — but it also means
the bytes are meaningless on their own. Decode them with the *wrong* schema (say,
two fields swapped) and Avro cannot recover: it mis-slices the stream and either
returns garbage or, as you will see, **raises `EOFError: Expected 1984 bytes`** —
the number `1984` was a person's birth year that the wrong schema read as a string
length. Protocol Buffers, by contrast, tags every field, so a reader that knows
only *some* of the tags still recovers the fields it knows.

## Provenance

*Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025),
Chapter 5 — *Encoding and Evolution*, §"Avro" and §"But what is the writer's
schema?":

> you go through the fields in the order that they appear in the schema

Because the parser walks the fields *in schema order* with nothing in the data to
re-synchronize on, it must have the exact schema the data was written with. Here
you encode a record, decode it with the right schema, then with a wrong one — and
watch the compactness turn into total unreadability.

## Prerequisites

The surprise is that "smallest" and "undecodable without the schema" are the *same*
fact. These help; each line notes where it shows up.

1. **Avro's encoding: values in schema order** — Avro writes each field's value
   back-to-back in the order the schema declares, with no names and no tags. This
   is why it is the smallest, and why step (b) breaks so badly.
2. **Writer's schema vs. reader's schema** — Avro decoding needs the schema the
   data was *written* with; a merely different reader schema is not enough. Step
   (b) hands the reader a different schema and it fails.
3. **Field tags (Protobuf)** — Protobuf prefixes each field with a small integer
   tag, so a field is self-locating in the byte stream. Step (c) relies on this to
   recover fields from a partial, renamed schema.
4. **Varint / length-prefix framing** — Avro strings are length-prefixed and longs
   are varints; when the schema is wrong, a *value* (1984) gets read as a *length*.
   That mix-up is exactly the `EOFError` in step (b).

### Where to learn them

- **Avro's encoding and the writer/reader-schema idea (#1, #2):** DDIA §"Avro" and
  §"But what is the writer's schema?"; the Avro specification's "Binary Encoding"
  and "Schema Resolution" sections.
- **Protobuf tags (#3):** DDIA §"Protocol Buffers"; the Protobuf "Encoding" page
  (wire types and field numbers).
- **Length-prefix / varint framing (#4):** the Avro and Protobuf encoding specs.

If only one is new, make it #1 — "values in schema order, nothing else" is the
whole surprise.

## Environment

Deterministic, so the bytes and the error reproduce exactly:

- macOS 26.5.2 (Darwin 25.5.0), arm64
- Python 3.12.13 via `uv`, with `fastavro` and Protocol Buffers (`grpcio-tools`
  compiling the `.proto` at runtime + the `protobuf` runtime) — no Docker
- One record: `{name: "Zhang Wei", birthYear: 1984, city: "Shanghai"}`

## Mental model

One record, two schema-based encodings, and what happens when the reader's schema
does not match the writer's:

- **Avro** writes `len + "Zhang Wei"`, then the varint `1984`, then `len +
  "Shanghai"` — a flat stream of values. To read it you replay the *same* schema
  field by field. Change the schema and every offset after the first mismatch is
  wrong; there is no tag to re-anchor on.
- **Protobuf** writes `(tag 1, "Zhang Wei") (tag 2, 1984) (tag 3, "Shanghai")`.
  Each field carries its own tag, so a reader can skip a tag it does not know and
  still locate the ones it does — even if a field was renamed or dropped from its
  schema.

We encode once, decode with the right schema, then with a wrong one, and contrast
with Protobuf's partial reader. The whole exercise is one script,
`code/avro_needs_schema.py`.

## Setup

```
cd study/ddia/ch05/code
uv run --python 3.12 --with fastavro --with grpcio-tools --with protobuf \
    python3 avro_needs_schema.py
```

Do **not** read the output yet — make each prediction first.

## Steps

### Step 1 — predict the size, and the wrong-schema decode

**Predict.** The record is `{name, birthYear, city}`. Guess (1) how big the Avro
encoding is versus Protobuf, and (2) what happens when you decode the Avro bytes
with a schema that has `city` and `birthYear` *swapped* — a clean wrong answer, an
exception, or something in between?

**Do.** Run the script.

**Observe.**

```
(a) Avro encoding of {name, birthYear, city} -- values only, in schema order:
    21 bytes: 12 5a 68 61 6e 67 20 57 65 69 80 1f 10 53 68 61 6e 67 68 61 69
    no field names, no tags -- just a length + "Zhang Wei", a varint 1984, a length + "Shanghai".
```

**21 bytes** — smaller than Protobuf's 24 (below), because Avro spends nothing on
tags. The hex is just: `12` (length 9, zigzag) + `Zhang Wei`, then `80 1f` (the
varint for 1984), then `10` (length 8) + `Shanghai`. Values, nothing else.

### Step 2 — decode with the right schema, then the wrong one

**Observe.**

```
(b) Decode those same bytes with the WRITER's schema (name, birthYear, city):
    -> {'name': 'Zhang Wei', 'birthYear': 1984, 'city': 'Shanghai'}   (correct)

    Now decode the SAME bytes with a different schema -- city and birthYear swapped
    (to Avro: "string, string, long" instead of "string, long, string"):
    -> RAISED EOFError: Expected 1984 bytes, read 9
    The bytes carry no field boundaries, so a wrong schema cannot even be applied. Either way:
    without the exact writer's schema, the data is unreadable.
```

Look at the error: **`Expected 1984 bytes`**. The wrong schema read `name`
correctly, then — expecting `city` (a string) next — grabbed the varint `1984`
(the *birth year*) and interpreted it as a string *length*. It then tried to read
1984 bytes of city name from a buffer with 9 left, and blew up. The value became a
length because nothing in the bytes says "this is a number, that is a string." A
wrong schema does not give you a wrong-but-plausible record; it desynchronizes the
whole stream.

### Step 3 — Protobuf recovers known fields from a partial schema

**Predict.** Encode the same record with Protobuf and decode it with a schema that
knows only tags 1 and 3, has *renamed* field 3 `city` → `town`, and *dropped* the
`birthYear` field entirely. What comes back?

**Observe.**

```
(c) Protobuf encoding of the same record: 24 bytes: 0a 09 5a 68 61 6e 67 20 57 65 69 10 c0 0f 1a 08 53 68 61 6e 67 68 61 69
    Each field is a (tag, value) pair. Decode with a PARTIAL schema that knows only
    tag 1 and tag 3 -- and has renamed tag 3's field city -> town, and dropped tag 2:
    -> {'name': 'Zhang Wei', 'town': 'Shanghai'}
       tag 2 (birthYear) was unknown, but preserved: re-serialize and a full reader still gets 1984.
    The tags made name and city self-locating despite the rename and the missing field.
```

The partial reader gets `name` and `town` (the renamed `city`) **correctly** — the
name in the `.proto` never touched the wire; only tag 3 did. Tag 2 (`birthYear`)
was unknown to this schema, but proto3 *preserves* unknown fields: re-serialize
the parsed message and a full reader still reads back `1984`. Tags let a reader
skip what it does not understand and still nail what it does.

### Step 4 — the consequence

**Observe.**

```
(d) So Avro must SHIP the writer's schema alongside the data:
    - an Avro object-container file writes the schema once, in its header;
    - a database or stream stores a schema version/ID and looks it up in a registry.
    The bytes alone are undecodable -- that is the price of being the smallest.
```

## What you should see

- Avro is the smallest: **21 bytes** vs Protobuf's **24**, because it stores no
  tags and no names.
- Decoding Avro with a *swapped* schema does not misread quietly — it **raises
  `EOFError: Expected 1984 bytes`**, the birth-year value mistaken for a string
  length.
- Protobuf's partial reader still recovers `name` and the renamed `city`→`town`,
  and preserves the unknown `birthYear` (1984) across a round-trip.

## Why

Avro's compactness and its fragility are one and the same design choice. Because a
schema fixes the field *order*, Avro can drop everything except the values and
write them back to back — reaching the floor of "just the data," a byte smaller
than Protobuf here and much smaller than JSON. But that leaves the byte stream with
no internal structure: no field names, no tags, no type markers, no lengths except
the ones values carry themselves. The only thing that says "the first value is a
string of length 9, the next is a long, the next is a string" is the schema, held
*outside* the data. Replay the correct schema and every field lands on its correct
bytes. Replay a schema whose types or order differ, and the very first mismatch
throws every subsequent offset off — a `long` read as a `string` length, a value
read as a count — so you get an exception or a plausible-looking lie, never a
reliable record.

Protobuf makes the opposite trade for one extra byte. Every field is prefixed with
a tag that encodes both its field number and its wire type, so each field is
*self-locating*: a reader can walk the stream, and for each field either decode it
(known tag) or skip exactly the right number of bytes (unknown tag). That is why a
partial, renamed schema still works — the field name in the `.proto` is a local
label, but the tag on the wire is the real identity. It is also why Protobuf can
evolve gracefully: add a field with a new tag and old readers skip it; the readers
never had to see the writer's exact schema, only agree on tag numbers.

**The boundary — Avro is not "broken," it just moves the schema out of band.** The
`EOFError` is not a bug; it is Avro refusing to guess without the contract it was
built to rely on. In real use you never hand Avro a random schema: an Avro
*object-container file* writes the writer's schema once in its header, so a whole
file of millions of records pays the schema cost a single time; a database column
or a Kafka topic stores a small schema *version/ID* with each record and resolves
it against a **schema registry**. Given the writer's schema *and* a reader's
schema, Avro's schema-resolution rules then reconcile compatible differences
(added fields with defaults, reordering by name) — the power the book highlights.
The failure you saw is what happens when that out-of-band schema is *absent or
wrong*: the bytes alone, unlike Protobuf's, carry nothing to fall back on. Smallest
on the wire, most dependent on the schema — the same coin, both sides.

### Go deeper

1. **Make it garbage instead of an error.** Swap two *string* fields (both same
   type) instead of a string and a long, and re-decode. Predict whether Avro now
   returns a clean-but-wrong record (values in the wrong keys) rather than raising
   — and why same-type swaps desynchronize *less* violently than type mismatches.
2. **Give Avro the reader's *and* writer's schema.** Use `fastavro`'s reader with
   both schemas (schema resolution) and add a field with a default to the reader.
   Predict what resolves cleanly (new field filled from default) versus what still
   cannot (a reordered field of a different type).
3. **Ship the schema in the bytes.** Re-encode with `fastavro.writer` to an
   object-container file and inspect the header. Predict how many records it takes
   before the once-per-file schema is cheaper than Protobuf's per-record tags.