# DDIA Ch. 5: the field-tag landmine — reuse a tag, silently corrupt

## Concept

Protocol Buffers identifies each field by its tag *number*, not its name — the
name lives only in the schema, and the wire bytes carry `(tag, value)` pairs. So
if a new schema version reuses an old tag number for a *different* field of the
same wire type, old encoded bytes are silently reinterpreted as the new field:
no error, wrong meaning. You will encode an event's `timestamp_ms` under tag 2,
then decode those same bytes under a v2 schema that reuses tag 2 for `user_id`,
and watch **`1719950400000`** — an epoch-millisecond timestamp — come back as a
plausible-looking user id. The safe evolution is a *fresh* tag number.

## Provenance

*Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025),
Chapter 5 — *Encoding and Evolution*, §"Field tags and schema evolution":

> you cannot change a field's tag, since that would make all existing encoded
> data invalid.

The book states the rule; here you break it on purpose and watch exactly *how*
the data goes bad — silently, with no error, into a plausible wrong value.

## Prerequisites

The surprise is that a schema change compiles and decodes cleanly while meaning
something entirely different. These help; each line notes where it shows up.

1. **Field tags identify fields, names don't** — Protobuf's wire format is a
   sequence of `(tag number, value)` pairs; the field *name* exists only in the
   `.proto`. This is the whole mechanism: tag 2 is tag 2 regardless of what you
   call it (v1 `timestamp_ms` vs v2 `user_id`).
2. **Wire types** — the low 3 bits of each tag byte encode a wire type (0 =
   varint for `int64`, 2 = length-delimited for `string`). The decoder uses this
   to know how many bytes the value spans — and, in v2c, to notice a mismatch.
3. **proto3 decoding of missing / unknown fields** — an absent field takes its
   zero default (v2b's `user_id = 0`); a tag whose wire type doesn't match the
   schema is treated as unknown and skipped (v2c's dropped `string`).
4. **The record model** — one message, one `int64` field on tag 2, so only the
   *tag reuse* varies between versions and nothing else can explain the result.

### Where to learn them

- **Field tags and wire format (#1, #2):** DDIA §"Field tags and schema
  evolution"; the Protobuf "Encoding" spec page (the `(field_number << 3) |
  wire_type` key byte).
- **Default and unknown fields (#3):** the Protobuf "Proto3 Language Guide"
  (default values) and the encoding page (unknown fields).

If only one is new, make it #1 — that tags, not names, identify fields is the
entire landmine.

## Environment

Deterministic, so the bytes reproduce exactly:

- macOS 26.5.2 (Darwin 25.5.0), arm64
- Python 3.12.13 via `uv`, with Protocol Buffers (`grpcio-tools` compiling each
  `.proto` at runtime + the `protobuf` runtime) — no Docker
- One event value: `timestamp_ms = 1719950400000` (epoch-ms, 2024-07-02T20:00:00Z)

## Mental model

One event, encoded once under v1, then decoded under three v2 schemas:

- **v1** — `message Event { int64 timestamp_ms = 2; }`. Encode the timestamp; the
  wire bytes are `(tag 2, varint value)`.
- **v2 (dangerous)** — `message Event { int64 user_id = 2; }`. Reuses tag 2 for a
  different `int64`. Same wire type, so the old value decodes cleanly — as a
  `user_id`.
- **v2b (safe)** — `message Event { int64 timestamp_ms = 2; int64 user_id = 3; }`.
  `user_id` gets a *fresh* tag 3. Old bytes keep the timestamp; `user_id`
  defaults to 0.
- **v2c (wrong type)** — `message Event { string user_id = 2; }`. Reuses tag 2 but
  as a `string` (different wire type), so the old int is a wire-type mismatch.

We decode *the same old bytes* under each and print the result. The whole
exercise is one script, `code/field_tag_landmine.py`.

## Setup

```
cd study/ddia/ch05/code
uv run --python 3.12 --with grpcio-tools --with protobuf \
    python3 field_tag_landmine.py
```

Do **not** read the output yet — make each prediction first.

## Steps

### Step 1 — encode under v1

**Predict.** `timestamp_ms = 1719950400000` is encoded under `int64 ... = 2`.
Roughly how many bytes, and what does the *first* byte encode?

**Do.** Run the script and read the first block.

**Observe.**

```
v1 schema:  message Event { int64 timestamp_ms = 2; }
  encode timestamp_ms = 1719950400000  (epoch-ms, 2024-07-02T20:00:00Z)
  wire bytes (7):  10 80 b4 a1 a8 87 32
  the leading 0x10 is (tag=2, wire type 0 = varint); the rest is the value.
```

Seven bytes: one tag byte plus a six-byte varint. `0x10` is `(2 << 3) | 0` — tag
number 2, wire type 0 (varint). Note what is *not* here: the string
`"timestamp_ms"` appears nowhere. The name lived only in the schema.

### Step 2 — decode under v2 (reuse the tag)

**Predict.** v2 reuses tag 2 for `int64 user_id`. Decode the *same* v1 bytes.
Does it raise an error? If not, what is `user_id`?

**Do.** Read the v2 block.

**Observe.**

```
v2 schema:  message Event { int64 user_id = 2; }   <-- REUSED tag 2
  decode the SAME old bytes with v2:
    REUSED TAG -> user_id = 1719950400000   <-- wrong value, no error
    (that is the timestamp 1719950400000 silently read as a user id)
```

No error. The bytes still say "tag 2, this varint," and v2 says "tag 2 is
`user_id`," so a timestamp becomes a user id of 1.7 trillion. **This is the
landmine**: it compiles, it decodes, it is wrong.

### Step 3 — decode under v2b (fresh tag) and v2c (wrong type)

**Predict.** v2b keeps `timestamp_ms` on tag 2 and adds `user_id` on a *fresh*
tag 3. v2c reuses tag 2 but as a `string`. What does each do to the old bytes?

**Do.** Read the last two blocks.

**Observe.**

```
v2b schema: message Event { int64 timestamp_ms = 2; int64 user_id = 3; }
  decode the SAME old bytes with v2b (fresh tag 3 for user_id):
    FRESH TAG  -> timestamp_ms = 1719950400000   <-- correct, preserved
                  user_id      = 0            <-- absent -> default 0

v2c schema: message Event { string user_id = 2; }   <-- REUSED tag 2, new wire type
  decode the SAME old bytes with v2c (int bytes read as a string):
    WRONG TYPE -> no error; user_id = ''   <-- field silently DROPPED
    (the wire types differ (varint vs length-delimited), so protobuf
     treats tag 2 as an unknown field and skips it -- data lost, not misread)

takeaway: reuse a tag with the SAME wire type -> SILENT corruption, a
          plausible wrong value with no error (v2, the landmine).
          Reuse it with a DIFFERENT wire type -> still no error, but the
          value is silently dropped (v2c). A FRESH tag is the only safe
          evolution (v2b): old data keeps its meaning, new field defaults.
```

The fresh tag is the fix: the old timestamp keeps its meaning on tag 2, and
`user_id` — never present in the old bytes — reads as its zero default. The
wrong-type reuse (v2c) is the interesting middle case: it *also* raises no error,
but because the wire types differ, protobuf can tell the byte isn't a valid
`string` field and silently *drops* it rather than handing you a plausible wrong
value.

## What you should see

- v1 encodes to **7 bytes**: `10 80 b4 a1 a8 87 32`, a tag byte plus a varint —
  and no field name anywhere.
- Reusing tag 2 for a same-type field (v2) **silently misreads** the timestamp as
  `user_id = 1719950400000`, with no error.
- A fresh tag (v2b) **preserves** `timestamp_ms = 1719950400000` and defaults the
  new `user_id` to `0`.
- Reusing tag 2 as a different wire type (v2c) raises no error either, but
  **silently drops** the field (`user_id = ''`).

## Why

Protobuf's wire format is deliberately minimal: each field is a key byte —
`(field_number << 3) | wire_type` — followed by the value, and nothing else. The
field *name* is compile-time-only; it never travels. That is exactly what makes
Protobuf compact (the size-ladder exercise), and it is exactly what makes tag
reuse dangerous. The decoder does not — cannot — check that "the field on tag 2
used to be called `timestamp_ms`." It reads tag 2, looks up tag 2 in *its*
schema, finds `user_id`, and assigns the value. The bytes are valid; only the
*agreement* about what tag 2 means has been broken, and Protobuf has no way to
detect a broken agreement.

The wire type is the one consistency check the format offers, and it is a check
about *shape*, not *meaning*. In v2 the shapes match — both `int64` are varints —
so nothing looks wrong and the value flows straight into the wrong field. In v2c
the shapes differ — a varint is not a length-delimited string — so the decoder
notices tag 2 doesn't fit and treats it as an unknown field, skipping it. Neither
case errors; the difference is whether you get a plausible wrong value or a
missing one. Both are silent, and silent is the hazard.

**The boundary — the wire type is a shape check, not a meaning check, so
same-type tag reuse is the only truly silent corruption.** The compiler will not
save you: v2 is valid `.proto` and valid Protobuf. What saves you is process —
never reuse a tag number, and mark retired tags `reserved` so the compiler
*refuses* to reassign them. Note the failure modes rank by danger: same wire type
(v2) gives a believable wrong value and is worst; different wire type (v2c)
loses the data but at least won't hand you garbage that looks real; a truly fresh
tag (v2b) is the only forward- and backward-compatible move. And this is a
Protobuf-shaped hazard specifically — Avro has no tags at all and instead depends
on the exact writer schema (the Avro exercise), trading tag-reuse for
schema-resolution as its evolution mechanism.

### Go deeper

1. **`reserved` as the guardrail.** Change v2 to `message Event { reserved 2;
   int64 user_id = 3; }` and try to compile a version that puts a field back on
   tag 2. Predict whether `protoc` errors — and why `reserved` is the real fix
   the book recommends over "just remember not to reuse it."
2. **Truncated vs. mismatched varints.** Feed v2 a *truncated* copy of the old
   bytes (drop the last byte). Predict whether it errors, defaults, or yields a
   different wrong number — and connect that to why wire type alone can't
   validate meaning.
3. **Adding vs. removing a required-ish field.** Encode under v2b (both fields
   set), then decode under v1 (which only knows tag 2). Predict what happens to
   the tag-3 `user_id` bytes — this is the *forward*-compatibility direction, the
   mirror of Step 3.