Avro stores bare values in schema order — no names, no tags — so it is the most compact encoding (21 bytes vs Protobuf's 24) and the bytes are meaningless on their own. Decode with the wrong schema and Avro raises EOFError: Expected 1984 bytes — a birth year read as a string length. Protobuf's tags recover known fields even from a partial, renamed schema.
Avro strips a record down to bare values in schema order: no field names, no tags, not even a boundary between one field and the next. That makes it the most compact of the binary encodings — here 21 bytes to Protobuf's 24 — but it also means the bytes are meaningless on their own. Decode them with the wrong schema (say, two fields swapped) and Avro cannot recover: it mis-slices the stream and either returns garbage or, as you will see, raises EOFError: Expected 1984 bytes — the number 1984 was a person's birth year that the wrong schema read as a string length. Protocol Buffers, by contrast, tags every field, so a reader that knows only some of the tags still recovers the fields it knows.
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 5 — Encoding and Evolution, §"Avro" and §"But what is the writer's schema?":
you go through the fields in the order that they appear in the schema
Because the parser walks the fields in schema order with nothing in the data to re-synchronize on, it must have the exact schema the data was written with. Here you encode a record, decode it with the right schema, then with a wrong one — and watch the compactness turn into total unreadability.
The surprise is that "smallest" and "undecodable without the schema" are the same fact. These help; each line notes where it shows up.
EOFError in step (b).uv, with fastavro and Protocol Buffers (grpcio-tools compiling the .proto at runtime + the protobuf runtime) — no Docker. One record: {name: "Zhang Wei", birthYear: 1984, city: "Shanghai"}.
One record, two schema-based encodings, and what happens when the reader's schema does not match the writer's:
len + "Zhang Wei", then the varint 1984, then len + "Shanghai" — a flat stream of values. To read it you replay the same schema field by field. Change the schema and every offset after the first mismatch is wrong; there is no tag to re-anchor on.(tag 1, "Zhang Wei") (tag 2, 1984) (tag 3, "Shanghai"). Each field carries its own tag, so a reader can skip a tag it does not know and still locate the ones it does — even if a field was renamed or dropped from its schema.We encode once, decode with the right schema, then with a wrong one, and contrast with Protobuf's partial reader. The whole exercise is one script, code/avro_needs_schema.py.
cd study/ddia/ch05/code
uv run --python 3.12 --with fastavro --with grpcio-tools --with protobuf \
python3 avro_needs_schema.pyDo not read the output yet — make each prediction first.
Predict. The record is {name, birthYear, city}. Guess (1) how big the Avro encoding is versus Protobuf, and (2) what happens when you decode the Avro bytes with a schema that has city and birthYear swapped — a clean wrong answer, an exception, or something in between?
Do. Run the script.
21 bytes — smaller than Protobuf's 24 (below), because Avro spends nothing on tags. The hex is just: 12 (length 9, zigzag) + Zhang Wei, then 80 1f (the varint for 1984), then 10 (length 8) + Shanghai. Values, nothing else.
Look at the error: Expected 1984 bytes. The wrong schema read name correctly, then — expecting city (a string) next — grabbed the varint 1984 (the birth year) and interpreted it as a string length. It then tried to read 1984 bytes of city name from a buffer with 9 left, and blew up. The value became a length because nothing in the bytes says "this is a number, that is a string." A wrong schema does not give you a wrong-but-plausible record; it desynchronizes the whole stream.
Predict. Encode the same record with Protobuf and decode it with a schema that knows only tags 1 and 3, has renamed field 3 city → town, and dropped the birthYear field entirely. What comes back?
The partial reader gets name and town (the renamed city) correctly — the name in the .proto never touched the wire; only tag 3 did. Tag 2 (birthYear) was unknown to this schema, but proto3 preserves unknown fields: re-serialize the parsed message and a full reader still reads back 1984. Tags let a reader skip what it does not understand and still nail what it does.
Avro moves the schema out of band: an object-container file writes it once in its header, and a database or stream stores a small version/ID resolved against a schema registry. The bytes on their own carry nothing to fall back on.
EOFError: Expected 1984 bytes, the birth-year value mistaken for a string length.name and the renamed city→town, and preserves the unknown birthYear (1984) across a round-trip.Avro's compactness and its fragility are one and the same design choice. Because a schema fixes the field order, Avro can drop everything except the values and write them back to back — reaching the floor of "just the data," a byte smaller than Protobuf here and much smaller than JSON. But that leaves the byte stream with no internal structure: no field names, no tags, no type markers, no lengths except the ones values carry themselves. The only thing that says "the first value is a string of length 9, the next is a long, the next is a string" is the schema, held outside the data. Replay the correct schema and every field lands on its correct bytes. Replay a schema whose types or order differ, and the very first mismatch throws every subsequent offset off — a long read as a string length, a value read as a count — so you get an exception or a plausible-looking lie, never a reliable record.
Protobuf makes the opposite trade for one extra byte. Every field is prefixed with a tag that encodes both its field number and its wire type, so each field is self-locating: a reader can walk the stream, and for each field either decode it (known tag) or skip exactly the right number of bytes (unknown tag). That is why a partial, renamed schema still works — the field name in the .proto is a local label, but the tag on the wire is the real identity. It is also why Protobuf can evolve gracefully: add a field with a new tag and old readers skip it; the readers never had to see the writer's exact schema, only agree on tag numbers.
EOFError is not a bug; it is Avro refusing to guess without the contract it was built to rely on. In real use you never hand Avro a random schema: an Avro object-container file writes the writer's schema once in its header, so a whole file of millions of records pays the schema cost a single time; a database column or a Kafka topic stores a small schema version/ID with each record and resolves it against a schema registry. Given the writer's schema and a reader's schema, Avro's schema-resolution rules then reconcile compatible differences (added fields with defaults, reordering by name) — the power the book highlights. The failure you saw is what happens when that out-of-band schema is absent or wrong: the bytes alone, unlike Protobuf's, carry nothing to fall back on. Smallest on the wire, most dependent on the schema — the same coin, both sides.
fastavro's reader with both schemas (schema resolution) and add a field with a default to the reader. Predict what resolves cleanly (new field filled from default) versus what still cannot (a reordered field of a different type).fastavro.writer to an object-container file and inspect the header. Predict how many records it takes before the once-per-file schema is cheaper than Protobuf's per-record tags.Sources: DDIA 2e, Ch. 5, §"Avro", §"But what is the writer's schema?", §"Protocol Buffers" · the Avro and Protobuf encoding specifications.