study › DDIA › Chapter 5 · Encoding and Evolution

The ID That Changed on the Way In

JSON never specifies number precision, so a parser may read every number as an IEEE-754 double — and JavaScript does. Send a 64-bit ID (a tweet snowflake) through such a parser and it comes back changed by 21, no error. A typed Protobuf int64 round-trips it exactly.

Concept

JSON has numbers, but it never says how precise they are — and it doesn't separate integers from floats. So a JSON parser is free to read every number as an IEEE-754 double, and JavaScript's JSON.parse does exactly that. A double holds only 53 bits of integer precision, so any integer above 2⁵³ loses its low bits on the way in. You will send a perfectly ordinary 64-bit ID — a tweet snowflake — through such a parser and watch it come back changed by 21, no error raised. A typed encoding (Protobuf int64) round-trips the same ID exactly.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 5 — Encoding and Evolution, §"JSON, XML, and Binary Variants":

integers greater than 2⁵³ cannot be exactly represented in an IEEE 754 double-precision floating-point number, so such numbers become inaccurate when parsed in a language that uses floating-point numbers, such as JavaScript.

The book names the exact failure and the exact victim (X/Twitter, whose IDs are 64-bit). Here you reproduce it and see the off-by-21.

Prerequisites

The surprise is about how numbers survive (or don't) a text format. These help; each line notes where it shows up.

IEEE-754 double precision — a 64-bit float stores a 52-bit mantissa, giving exactly 53 bits of integer precision. Above 2⁵³ = 9,007,199,254,740,992, not every integer is representable, so some round to a neighbor.
JSON's number model — JSON has one numeric type and no precision rule, so a conforming parser MAY store numbers as doubles. The format doesn't lose the digits; a double-based parser does.
64-bit IDs — snowflake IDs (X/Twitter, Discord) are int64, routinely above 2⁵³. This is the real-world blast radius.
Schema-typed integers — Protobuf/Avro declare a field int64/long, so the decoder never guesses "double"; the integer is preserved by construction.

Where to learn the prerequisites Float precision (#1): search "why 2^53 floating point integer" — or read IEEE-754 basics; the mantissa width is the whole story. JSON number pitfalls (#2–#3): DDIA §"JSON, XML, and Binary Variants"; the X/Twitter developer docs on id vs id_str. If only one is new, make it #1 — 53 bits of precision is the entire mechanism.

Environment these numbers came from Deterministic, so the numbers reproduce exactly. Captured on macOS 26.5.2 (Darwin 25.5.0), arm64, Python 3.12.13 via uv; Protocol Buffers via grpcio-tools + protobuf (compiled at runtime) — no Docker. One 64-bit ID: 1234567890123456789.

Mental model

We take a 64-bit ID (1234567890123456789, above 2⁵³) and:

put it through the IEEE-754 boundary directly (int(float(n))) to see where precision ends;
serialize it to JSON text and parse it back with a parser that reads numbers as doubles — mimicking JavaScript's JSON.parse, in Python via json.loads(..., parse_int=float);
round-trip it through a Protobuf int64 field for contrast.

The whole exercise is one script, code/json_integer_precision.py.

Setup

cd study/ddia/ch05/code
uv run --with grpcio-tools --with protobuf python3 json_integer_precision.py

Do not read the output yet — make each prediction first.

Step 1 · the precision boundary (calibration)

Predict. A double has 53 bits of integer precision. What happens to 2⁵³ and 2⁵³ + 1 when forced through a double?

What happens to 2^53 and 2^53 + 1 as doubles?

the IEEE-754 double boundary (a double has 53 bits of integer precision): 2^53 = 9007199254740992 -> as a double -> 9007199254740992 (exact) 2^53 + 1 = 9007199254740993 -> as a double -> 9007199254740992 (off by 1)

2⁵³ survives; 2⁵³ + 1 cannot be represented and rounds down to 2⁵³ — the very first integer a double can't hold. Above this line, integers start collapsing onto their neighbors.

Step 2 · round-trip the ID through JSON (the surprise)

Predict. Send 1234567890123456789 as JSON and parse it back with a double-based parser (what a browser does). Does the ID come back unchanged?

Does the 64-bit ID survive a double-based JSON parse?

round-tripping the ID through JSON: JSON text sent: {"id": 1234567890123456789} parsed as doubles (JavaScript, or json.loads parse_int=float): got: 1234567890123456768 off by: 21 <- silently wrong; the ID no longer matches (Python's DEFAULT json happens to keep it: True -- but JSON doesn't guarantee this, and JavaScript's JSON.parse does not.)

Off by 21. The ID came back ...456768 instead of ...456789 — a different record, and no error anywhere. Note the parenthetical: Python's default parser happens to preserve it (Python uses big integers), but that's luck, not a guarantee — the same bytes fed to JavaScript's JSON.parse are mangled, and you rarely control the receiver's parser.

Step 3 · the typed fix

Predict. Encode the same ID in a Protobuf int64 field and decode it. Same problem, or exact?

Does a typed int64 round-trip the ID?

the typed fix -- Protobuf int64 round-trips exactly: sent 1234567890123456789 -> decoded 1234567890123456789 (exact: True) which is why X/Twitter's API returns the ID twice: id (number) and id_str (string).

Exact. Because the schema declares the field int64, the decoder never reaches for a double — the integer is preserved by construction. And the famous real-world workaround for JSON: X/Twitter returns every ID as both a number (id) and a string (id_str), because a string of digits sails through any JSON parser untouched.

What you should see

2⁵³ + 1 rounds down to 2⁵³ through a double — the first unrepresentable integer.
The 64-bit ID round-tripped via a double-based JSON parser comes back off by 21 — silently wrong, no error.
Protobuf int64 round-trips the ID exactly; strings (id_str) are the JSON-side fix.

Why

A JSON number is just text like 1234567890123456789, and the format says nothing about how a reader should store it. JavaScript — where JSON was born — has exactly one number type, the IEEE-754 double, so JSON.parse turns that text into a double. A double devotes 52 bits to the mantissa, so it can represent every integer up to 2⁵³ and then only even ones for a while, then only multiples of 4, and so on: past 2⁵³ the representable integers thin out, and any value that isn't one of them snaps to the nearest that is. 1234567890123456789 isn't representable, so it lands on 1234567890123456768 — 21 away. Nothing throws; the parser did exactly what the spec allows.

The deeper point is that JSON pushes the integer/float and precision decisions onto the reader, so correctness depends on a parser you often don't own. Python's json preserves big integers, but that's an implementation choice, not a JSON guarantee — swap in a browser, a JavaScript service, or a library that parses numbers as floats, and the same bytes silently corrupt. A schema-based encoding removes the guessing: declaring a field int64 (Protobuf) or long (Avro) tells every decoder to keep 64 bits, so the value round-trips regardless of the language. Where you're stuck with JSON, you dodge the issue by not sending the number as a number at all — hence id_str, an ID encoded as a string.

The boundary — it's an integer-size-and-parser problem, not "JSON is broken" JSON is fine for numbers that fit in 53 bits (most counts, prices, small IDs) and for anything you send as a string. The danger is precisely large integers — 64-bit IDs, nanosecond timestamps, big counters — decoded by a double-based parser. Three escapes, any of which works: send the value as a JSON string; use a JSON parser that preserves big integers (and control the whole path, which you usually can't); or use a typed binary encoding whose schema pins the width. The failure is silent and data-dependent, so it slips through tests built on small numbers and surfaces in production the day an ID crosses 2⁵³.

Go deeper

Find the first broken ID. Binary-search for the smallest integer above 2⁵³ that a double round-trip changes. Predict it (hint: 2⁵³ + 1) and confirm.
Strings survive. Send the ID as {"id_str": "1234567890123456789"} and parse it back. Predict whether the string round-trips exactly through the double-based parser (it should) — the id_str fix in one line.
Timestamps, too. Nanosecond epoch timestamps passed 2⁵³ years ago (in nanoseconds). Encode time.time_ns() as a JSON number through the double-parser and predict how many nanoseconds of precision you lose.

Sources: DDIA 2e, Ch. 5, §"JSON, XML, and Binary Variants" · IEEE-754 double precision · the X/Twitter API's id vs id_str.