Add one field and two migrations must hold at once — a v1 reader silently ignores a new Protobuf field (forward), a v2 reader fills a missing one with a default (backward). Then add an Avro field without a default and watch the read fail with SchemaResolutionError. "Just add a field" is safe or catastrophic depending on one word: default.
Add one field to a schema and two migrations must keep working at once: old code has to read data written by new code (forward compatibility), and new code has to read data written by old code (backward compatibility). Protobuf and Avro both give you this — but only under a rule that is easy to break. You will add an email field to a Protobuf message and watch a v1 reader silently ignore it (forward) while a v2 reader fills the missing value with a default (backward). Then you'll add a field to an Avro record twice: once with a default — a new reader reads old bytes fine — and once without — and watch the read fail with an exception. The surprise is that "just add a field" is safe or catastrophic depending on one word: default.
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 5 — Encoding and Evolution, §"Field tags and schema evolution" (Protobuf) and §"Schema evolution rules" (Avro):
you can only add a field if it has a default value
The book states the rule; here you make a reader obey it — and then break it and read the exact exception the runtime throws.
The surprise is about which reader meets which bytes. These help; each line notes where it shows up.
email (tag 3)."", 0); Avro requires you to declare a default in the schema, and refuses to invent one. This distinction is the whole Avro half of the exercise.uv, with Protocol Buffers (grpcio-tools compiling the .proto files at runtime + the protobuf runtime) and Avro (fastavro) — no Docker. Two Protobuf schema versions (Person v1 without email, v2 with it) and two Avro reader schemas (one adds favoriteColor with a default, one without).
One record, {userName: "Martin", id: 1337}, and a fourth field added on top:
email set; the v1 reader has no field for tag 3, so it skips the unknown tag and returns the fields it knows.email; the v2 reader sees the field is absent and hands back the type default for a string, "".favoriteColor; the reader schema adds it with default: "blue", so resolution fills the value from the schema.SchemaResolutionError.The whole exercise is one script, code/schema_evolution.py, which compiles both .proto versions at runtime and runs all four cases.
cd study/ddia/ch05/code
uv run --python 3.12 --with grpcio-tools --with protobuf --with fastavro \
python3 schema_evolution.pyDo not read the output yet — make each prediction first.
Predict. v2 encodes a Person with email set; you hand those bytes to a v1 reader that has never heard of email. Does ParseFromString throw, drop the record, or return the fields it recognizes? What happens to the email bytes?
Do. Run the script; read the === PROTOBUF === block.
The v1 reader succeeds and returns user_name and id. It met tag 3, had no field for it, and skipped it — the email bytes are simply ignored. Adding a field did not break old readers.
Predict. Now the reverse: v1 encodes without email, and a v2 reader that does have an email field decodes it. What value does email hold?
The v2 reader succeeds; email comes back as ''. Protobuf hands every absent field its type default — "" for a string — so new code reads old data without a hitch. Both directions work, for free.
Predict. The writer schema is {userName, id}. A new reader schema adds favoriteColor. First with default: "blue", then without any default. Old bytes (written before favoriteColor existed) don't carry the field. Which reader succeeds, and what does the other one do?
Same added field, opposite outcomes. With a default, Avro resolves the old bytes and fills favoriteColor from the schema. Without one, Avro has nowhere to get the value and raises SchemaResolutionError — the exact failure the rule warns about.
email silently dropped.email == '' (the string type default).favoriteColor == 'blue' filled from the schema.SchemaResolutionError: No default value for field favoriteColor.Compatibility comes down to two questions: what does a reader do with a field it doesn't know, and what does it do with a field it expects but that isn't there? Protobuf answers both structurally. Every proto3 field is optional and identified by a numeric tag, so a reader that meets an unknown tag can length-skip it without understanding it — that is forward compatibility, and it is why the v1 reader shrugged off email. And every field a reader expects but doesn't find on the wire gets a fixed type default ("", 0, empty list) — that is backward compatibility, and it is why the v2 reader turned a missing email into ''. Neither direction needs the schemas to match; they only need shared tag numbers.
Avro drops the tags — fields are matched by name during schema resolution — so it cannot fall back on a type default the way Protobuf does. When the reader's schema has a field the writer's schema lacked, Avro looks for a value and finds nothing on the wire. Its only escape is a default you declared in the reader schema. Provide one and resolution fills it; omit it and resolution has genuinely no value to produce, so it raises rather than guess. That is why the book's rule is not advice but a hard constraint: you can only add (or remove) a field if it has a default value.
default that lets new readers read old data (backward) is also what lets old readers read new data (forward): to remove a field safely it must have had a default, so that a reader still expecting it can fill the gap. So "add/remove only defaulted fields" is really one rule serving both directions. But defaults don't cover every change: renaming a field breaks Avro name-matching (use aliases), reusing a Protobuf tag number silently corrupts data (never recycle tags), and changing a field's type can fail even with a default. And none of this touches semantic evolution — if new code writes email but old code that never sets it keeps overwriting the record, the field can be silently dropped on a round-trip. Wire compatibility is necessary, not sufficient.
email to tag 2 (colliding with id) instead of 3, re-encode, and read with v1. Predict whether it errors or silently misinterprets the bytes — this is why you never reuse a tag number."" and a field never set both read back as "". Encode both, compare the bytes, and predict when that ambiguity matters (hint: optional / field presence).Sources: DDIA 2e, Ch. 5, §"Field tags and schema evolution", §"Schema evolution rules", §"The writer's schema and the reader's schema" · the Protobuf language guide and the Avro spec's "Schema Resolution".