# DDIA Ch. 5: schema evolution — old code reads new data, new code reads old

## Concept

Add one field to a schema and two migrations must keep working at once: *old*
code has to read data written by *new* code (**forward** compatibility), and
*new* code has to read data written by *old* code (**backward** compatibility).
Protobuf and Avro both give you this — but only under a rule that is easy to
break. You will add an `email` field to a Protobuf message and watch a v1 reader
silently ignore it (forward) while a v2 reader fills the missing value with a
default (backward). Then you'll add a field to an Avro record twice: once **with**
a default — a new reader reads old bytes fine — and once **without** — and watch
the read *fail with an exception*. The surprise is that "just add a field" is
safe or catastrophic depending on one word: `default`.

## Provenance

*Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini,
2025), Chapter 5 — *Encoding and Evolution*, §"Field tags and schema evolution"
(Protobuf) and §"Schema evolution rules" (Avro):

> you can only add a field if it has a default value

The book states the rule; here you make a reader obey it — and then break it and
read the exact exception the runtime throws.

## Prerequisites

The surprise is about which *reader* meets which *bytes*. These help; each line
notes where it shows up.

1. **Forward vs. backward compatibility** — forward = old code reads new data;
   backward = new code reads old data. Both are exercised below (Protobuf steps
   a and b), and it is easy to mix up which is which.
2. **Field tags (Protobuf)** — a field is identified by its integer tag, not its
   name; a reader that meets an unknown tag *skips* it, which is exactly why the
   v1 reader tolerates v2's `email` (tag 3).
3. **Type defaults vs. schema defaults** — Protobuf gives every absent field a
   built-in *type* default (`""`, `0`); Avro requires you to *declare* a
   `default` in the schema, and refuses to invent one. This distinction is the
   whole Avro half of the exercise.
4. **Writer schema vs. reader schema (Avro)** — Avro resolves bytes by pairing
   the schema they were *written* with against the schema you are *reading* with;
   the gap between them is where a missing default becomes fatal.

### Where to learn them

- **Forward/backward compatibility (#1):** DDIA §"Modes of dataflow" and the
  chapter intro; both terms are defined there before the format sections.
- **Field tags and defaults (#2–#3):** DDIA §"Field tags and schema evolution";
  the Protobuf language guide ("proto3" field presence and default values).
- **Writer/reader schema resolution (#4):** DDIA §"The writer's schema and the
  reader's schema"; the Avro spec's "Schema Resolution" section.

If only one is new, make it #3 — the presence or absence of a declared default
is the hinge the whole exercise turns on.

## Environment

Deterministic, so the transcript reproduces exactly:

- macOS 26.5.2 (Darwin 25.5.0), arm64
- Python 3.12.13 via `uv`, with Protocol Buffers (`grpcio-tools` compiling the
  `.proto` files at runtime + the `protobuf` runtime) and Avro (`fastavro`) — no
  Docker
- Two Protobuf schema versions (`Person` v1 without `email`, v2 with it) and two
  Avro reader schemas (one adds `favoriteColor` with a default, one without)

## Mental model

One record, `{userName: "Martin", id: 1337}`, and a fourth field added on top:

- **Protobuf, old reads new** — v2 encodes with `email` set; the v1 reader has no
  field for tag 3, so it *skips the unknown tag* and returns the fields it knows.
- **Protobuf, new reads old** — v1 encodes without `email`; the v2 reader sees the
  field is absent and hands back the *type default* for a string, `""`.
- **Avro, new reads old, field has a default** — the writer schema lacks
  `favoriteColor`; the reader schema adds it *with* `default: "blue"`, so
  resolution fills the value from the schema.
- **Avro, new reads old, field has no default** — same add, but *no* default; Avro
  has nothing to put there and raises `SchemaResolutionError`.

The whole exercise is one script, `code/schema_evolution.py`, which compiles both
`.proto` versions at runtime and runs all four cases.

## Setup

```
cd study/ddia/ch05/code
uv run --python 3.12 --with grpcio-tools --with protobuf --with fastavro \
    python3 schema_evolution.py
```

Do **not** read the output yet — make each prediction first.

## Steps

### Step 1 — Protobuf: old code reads new data (forward)

**Predict.** v2 encodes a `Person` with `email` set; you hand those bytes to a v1
reader that has never heard of `email`. Does `ParseFromString` throw, drop the
record, or return the fields it recognizes? What happens to the `email` bytes?

**Do.** Run the script; read the `=== PROTOBUF ===` block.

**Observe.**

```
=== PROTOBUF ===
v1: message Person { string user_name = 1; int64 id = 2; }
v2: message Person { string user_name = 1; int64 id = 2; string email = 3; }

[old-reads-new] v1 reader on v2 bytes (email set):  OK
    recovered: user_name='Martin', id=1337
    (email had tag 3, unknown to v1 -> silently ignored; v1 has no email field)
```

The v1 reader **succeeds** and returns `user_name` and `id`. It met tag 3, had no
field for it, and skipped it — the `email` bytes are simply ignored. Adding a
field did not break old readers.

### Step 2 — Protobuf: new code reads old data (backward)

**Predict.** Now the reverse: v1 encodes *without* `email`, and a v2 reader that
*does* have an `email` field decodes it. What value does `email` hold?

**Observe.**

```
[new-reads-old] v2 reader on v1 bytes (no email):   OK
    recovered: user_name='Martin', id=1337, email=''
    (email absent in old bytes -> string type default "" filled in)
```

The v2 reader **succeeds**; `email` comes back as `''`. Protobuf hands every
absent field its *type default* — `""` for a string — so new code reads old data
without a hitch. Both directions work, for free.

### Step 3 — Avro: new reads old, with a default vs. without

**Predict.** The writer schema is `{userName, id}`. A new reader schema adds
`favoriteColor`. First **with** `default: "blue"`, then **without** any default.
Old bytes (written before `favoriteColor` existed) don't carry the field. Which
reader succeeds, and what does the other one do?

**Observe.**

```
=== AVRO ===
writer schema : { userName: string, id: long }
reader schema : adds favoriteColor  (once WITH default "blue", once WITHOUT)

[new-reads-old, field HAS default] reader adds favoriteColor="blue":  OK
    recovered: {'userName': 'Martin', 'id': 1337, 'favoriteColor': 'blue'}
    (old bytes lack favoriteColor -> reader fills the schema default)

[new-reads-old, field has NO default] reader adds favoriteColor (required):
    FAILS -> SchemaResolutionError: No default value for field favoriteColor in Person
    (no default + absent in old bytes -> reader cannot supply the value)
```

**Same added field, opposite outcomes.** With a default, Avro resolves the old
bytes and fills `favoriteColor` from the schema. Without one, Avro has nowhere to
get the value and raises `SchemaResolutionError` — the exact failure the rule
warns about.

## What you should see

- **Protobuf forward** (old reads new): v1 reader on v2 bytes → **OK**, `email`
  silently dropped.
- **Protobuf backward** (new reads old): v2 reader on v1 bytes → **OK**, `email
  == ''` (the string type default).
- **Avro with a default**: new reader on old bytes → **OK**, `favoriteColor ==
  'blue'` filled from the schema.
- **Avro without a default**: new reader on old bytes → **`SchemaResolutionError:
  No default value for field favoriteColor`**.

## Why

Compatibility comes down to two questions: what does a reader do with a field it
*doesn't* know, and what does it do with a field it *expects* but that isn't
there? Protobuf answers both structurally. Every proto3 field is optional and
identified by a numeric tag, so a reader that meets an unknown tag can length-skip
it without understanding it — that is forward compatibility, and it is why the v1
reader shrugged off `email`. And every field a reader expects but doesn't find on
the wire gets a fixed *type* default (`""`, `0`, empty list) — that is backward
compatibility, and it is why the v2 reader turned a missing `email` into `''`.
Neither direction needs the schemas to match; they only need shared tag numbers.

Avro drops the tags — fields are matched by name during schema resolution — so it
cannot fall back on a type default the way Protobuf does. When the reader's schema
has a field the writer's schema lacked, Avro looks for a value and finds nothing
on the wire. Its only escape is a `default` you declared *in the reader schema*.
Provide one and resolution fills it; omit it and resolution has genuinely no value
to produce, so it raises rather than guess. That is why the book's rule is not
advice but a hard constraint: **you can only add (or remove) a field if it has a
default value.**

**The boundary — the rule cuts both ways, and defaults aren't the only trap.**
The same `default` that lets *new readers* read *old data* (backward) is also
what lets *old readers* read *new data* (forward): to remove a field safely it
must have had a default, so that a reader still expecting it can fill the gap. So
"add/remove only defaulted fields" is really one rule serving both directions.
But defaults don't cover every change: renaming a field breaks Avro name-matching
(use aliases), reusing a Protobuf tag number silently corrupts data (never
recycle tags), and *changing a field's type* can fail even with a default. And
none of this touches semantic evolution — if new code writes `email` but old code
that never sets it keeps overwriting the record, the field can be silently
dropped on a round-trip. Wire compatibility is necessary, not sufficient.

### Go deeper

1. **Break Protobuf on purpose.** Change v2's `email` to tag `2` (colliding with
   `id`) instead of `3`, re-encode, and read with v1. Predict whether it errors or
   silently misinterprets the bytes — this is why you never reuse a tag number.
2. **Remove a field, not just add one.** Make the writer schema the *bigger* one
   and the reader the *smaller* one (Avro), and predict whether dropping a field
   needs a default too. Then swap writer/reader to see forward vs. backward.
3. **Distinguish "absent" from "default."** In proto3 a field set to `""` and a
   field never set both read back as `""`. Encode both, compare the bytes, and
   predict when that ambiguity matters (hint: `optional` / field presence).