# DDIA Ch. 5: the ID that changed on the way in

## Concept

JSON has numbers, but it never says how *precise* they are — and it doesn't
separate integers from floats. So a JSON parser is free to read every number as
an IEEE-754 double, and JavaScript's `JSON.parse` does exactly that. A double
holds only 53 bits of integer precision, so any integer above 2⁵³ loses its low
bits on the way in. You will send a perfectly ordinary 64-bit ID — a tweet
snowflake — through such a parser and watch it come back **changed by 21**, no
error raised. A typed encoding (Protobuf `int64`) round-trips the same ID exactly.

## Provenance

*Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini,
2025), Chapter 5 — *Encoding and Evolution*, §"JSON, XML, and Binary Variants":

> integers greater than 2⁵³ cannot be exactly represented in an IEEE 754
> double-precision floating-point number, so such numbers become inaccurate when
> parsed in a language that uses floating-point numbers, such as JavaScript.

The book names the exact failure and the exact victim (X/Twitter, whose IDs are
64-bit). Here you reproduce it and see the off-by-21.

## Prerequisites

The surprise is about how numbers survive (or don't) a text format. These help;
each line notes where it shows up.

1. **IEEE-754 double precision** — a 64-bit float stores a 52-bit mantissa, giving
   exactly 53 bits of *integer* precision. Above 2⁵³ = 9,007,199,254,740,992, not
   every integer is representable, so some round to a neighbor.
2. **JSON's number model** — JSON has one numeric type and no precision rule, so a
   conforming parser MAY store numbers as doubles. The *format* doesn't lose the
   digits; a double-based *parser* does.
3. **64-bit IDs** — snowflake IDs (X/Twitter, Discord) are `int64`, routinely
   above 2⁵³. This is the real-world blast radius.
4. **Schema-typed integers** — Protobuf/Avro declare a field `int64`/`long`, so
   the decoder never guesses "double"; the integer is preserved by construction.

### Where to learn them

- **Float precision (#1):** search "why 2^53 floating point integer" — or read
  IEEE-754 basics; the mantissa width is the whole story.
- **JSON number pitfalls (#2–#3):** DDIA §"JSON, XML, and Binary Variants"; the
  X/Twitter developer docs on `id` vs `id_str`.

If only one is new, make it #1 — 53 bits of precision is the entire mechanism.

## Environment

Deterministic, so the numbers reproduce exactly:

- macOS 26.5.2 (Darwin 25.5.0), arm64
- Python 3.12.13 via `uv`; Protocol Buffers via `grpcio-tools` + `protobuf`
  (compiled at runtime) — no Docker
- One 64-bit ID: `1234567890123456789`

## Mental model

We take a 64-bit ID (`1234567890123456789`, above 2⁵³) and:

- put it through the IEEE-754 boundary directly (`int(float(n))`) to see where
  precision ends;
- serialize it to JSON text and parse it back with a parser that reads numbers
  as doubles — mimicking JavaScript's `JSON.parse`, in Python via
  `json.loads(..., parse_int=float)`;
- round-trip it through a Protobuf `int64` field for contrast.

The whole exercise is one script, `code/json_integer_precision.py`.

## Setup

```
cd study/ddia/ch05/code
uv run --with grpcio-tools --with protobuf python3 json_integer_precision.py
```

Do **not** read the output yet — make each prediction first.

## Steps

### Step 1 — the precision boundary (calibration)

**Predict.** A double has 53 bits of integer precision. What happens to `2⁵³`
and `2⁵³ + 1` when forced through a double?

**Observe.**

```
the IEEE-754 double boundary (a double has 53 bits of integer precision):
    2^53     = 9007199254740992   -> as a double -> 9007199254740992   (exact)
    2^53 + 1 = 9007199254740993   -> as a double -> 9007199254740992   (off by 1)
```

`2⁵³` survives; `2⁵³ + 1` **cannot be represented** and rounds down to `2⁵³` — the
very first integer a double can't hold. Above this line, integers start
collapsing onto their neighbors.

### Step 2 — round-trip the ID through JSON (the surprise)

**Predict.** Send `1234567890123456789` as JSON and parse it back with a
double-based parser (what a browser does). Does the ID come back unchanged?

**Observe.**

```
round-tripping the ID through JSON:
    JSON text sent:  {"id": 1234567890123456789}
    parsed as doubles (JavaScript, or json.loads parse_int=float):
        got:    1234567890123456768
        off by: 21    <- silently wrong; the ID no longer matches

    (Python's DEFAULT json happens to keep it: True -- but JSON
     doesn't guarantee this, and JavaScript's JSON.parse does not.)
```

**Off by 21.** The ID came back `...456768` instead of `...456789` — a different
record, and no error anywhere. Note the parenthetical: Python's *default* parser
happens to preserve it (Python uses big integers), but that's luck, not a
guarantee — the same bytes fed to JavaScript's `JSON.parse` are mangled, and you
rarely control the receiver's parser.

### Step 3 — the typed fix

**Predict.** Encode the same ID in a Protobuf `int64` field and decode it. Same
problem, or exact?

**Observe.**

```
the typed fix -- Protobuf int64 round-trips exactly:
    sent 1234567890123456789 -> decoded 1234567890123456789   (exact: True)

which is why X/Twitter's API returns the ID twice: id (number) and id_str (string).
```

**Exact.** Because the schema *declares* the field `int64`, the decoder never
reaches for a double — the integer is preserved by construction. And the famous
real-world workaround for JSON: X/Twitter returns every ID as both a number
(`id`) and a **string** (`id_str`), because a string of digits sails through any
JSON parser untouched.

## What you should see

- `2⁵³ + 1` rounds down to `2⁵³` through a double — the first unrepresentable
  integer.
- The 64-bit ID round-tripped via a double-based JSON parser comes back **off by
  21** — silently wrong, no error.
- Protobuf `int64` round-trips the ID **exactly**; strings (`id_str`) are the
  JSON-side fix.

## Why

A JSON number is just text like `1234567890123456789`, and the format says
nothing about how a reader should store it. JavaScript — where JSON was born —
has exactly one number type, the IEEE-754 double, so `JSON.parse` turns that text
into a double. A double devotes 52 bits to the mantissa, so it can represent
every integer up to 2⁵³ and then only *even* ones for a while, then only
multiples of 4, and so on: past 2⁵³ the representable integers thin out, and any
value that isn't one of them snaps to the nearest that is. `1234567890123456789`
isn't representable, so it lands on `1234567890123456768` — 21 away. Nothing
throws; the parser did exactly what the spec allows.

The deeper point is that JSON pushes the integer/float and precision decisions
onto the *reader*, so correctness depends on a parser you often don't own.
Python's `json` preserves big integers, but that's an implementation choice, not
a JSON guarantee — swap in a browser, a JavaScript service, or a library that
parses numbers as floats, and the same bytes silently corrupt. A schema-based
encoding removes the guessing: declaring a field `int64` (Protobuf) or `long`
(Avro) tells every decoder to keep 64 bits, so the value round-trips regardless
of the language. Where you're stuck with JSON, you dodge the issue by not sending
the number *as* a number at all — hence `id_str`, an ID encoded as a string.

**The boundary — it's an integer-size-and-parser problem, not "JSON is broken."**
JSON is fine for numbers that fit in 53 bits (most counts, prices, small IDs) and
for anything you send as a string. The danger is precisely large integers — 64-bit
IDs, nanosecond timestamps, big counters — decoded by a double-based parser.
Three escapes, any of which works: send the value as a JSON string; use a JSON
parser that preserves big integers (and control the whole path, which you usually
can't); or use a typed binary encoding whose schema pins the width. The failure
is silent and data-dependent, so it slips through tests built on small numbers
and surfaces in production the day an ID crosses 2⁵³.

### Go deeper

1. **Find the first broken ID.** Binary-search for the smallest integer above
   2⁵³ that a double round-trip changes. Predict it (hint: 2⁵³ + 1) and confirm.
2. **Strings survive.** Send the ID as `{"id_str": "1234567890123456789"}` and
   parse it back. Predict whether the string round-trips exactly through the
   double-based parser (it should) — the `id_str` fix in one line.
3. **Timestamps, too.** Nanosecond epoch timestamps passed 2⁵³ years ago (in
   nanoseconds). Encode `time.time_ns()` as a JSON number through the
   double-parser and predict how many nanoseconds of precision you lose.