study › DDIA › Chapter 3 · Data Models and Query Languages

The Ratings Matrix That's 99% Empty

A realistic 1%-full users×movies ratings matrix takes 200 MB stored densely and 4 MB stored sparsely — 50× smaller — because it pays only for the ratings that exist. The saving is exactly 1 / (2 × density). At Netflix scale, dense is 34 GB and won't fit in RAM; sparse is under 1 GB.

Concept

The book turns a table of movie ratings into a users × movies matrix (Figure 3-9) so linear algebra can drive recommendations. But real ratings matrices are almost entirely empty — a user rates a tiny fraction of all movies — so storing a number for every cell wastes nearly everything. You will hold a realistic 1%-full ratings matrix two ways and watch the dense array take 200 MB while the sparse representation takes 4 MB — 50× smaller — with a clean rule for exactly when sparse wins.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 3 — Data Models and Query Languages, §"DataFrames, Matrices, and Arrays":

the data in Figure 3-9 could be a part of a system for recommending movies … Once the data is in the form of a matrix of numbers, it is amenable to linear algebra operations, which form the basis of many ML algorithms.

The book shows the ratings-to-matrix transform. Here you measure the storage consequence the transform runs into — and the sparse representation that answers it.

Prerequisites

The surprise is about representing a mostly-empty matrix. These help; each line notes where it shows up.

Dense vs. sparse storage — dense stores a value for every cell; sparse stores only the nonzero entries plus where they are.
Density (fill fraction) — the share of cells that are nonzero. Recommender matrices are ~1% or less.
CSR (compressed sparse row) — a standard sparse format: three arrays — the nonzero values, their column indices, and per-row start pointers.
Cost per entry — dense costs one number per cell; CSR costs ~two numbers per nonzero (a value and a column index). That ratio sets the break-even.

Where to learn the prerequisites Sparse formats (#1, #3): the SciPy scipy.sparse docs (CSR/CSC/COO); any numerical-computing text on sparse matrices. Why it matters for ML (#2): DDIA §"DataFrames, Matrices, and Arrays" and the recommender-systems literature (the Netflix Prize matrix is the canonical example). If only one is new, make it #4 — the two-numbers-per-nonzero cost is the whole break-even.

Environment these numbers came from Deterministic seed, so the sizes reproduce exactly. Captured on macOS 26.5.2 (Darwin 25.5.0), arm64, with NumPy + SciPy on Python 3.12.13, run via uv — no Docker. A 10,000 × 5,000 ratings matrix at 1% density.

Mental model

A ratings matrix: rows are users, columns are movies, a cell holds a star rating if that user rated that movie and is empty otherwise. We build a 10,000 × 5,000 matrix (50,000,000 cells) at 1% density — like a real recommender (Netflix's was ~1.2%) — and store it two ways:

Dense — a float32 NumPy array with a number in every one of the 50M cells, rated or not.
Sparse (CSR) — only the ~500,000 actual ratings, each as a value plus a column index, with per-row pointers.

We compare nbytes. The whole exercise is one script, code/sparse_matrix.py.

Setup

cd study/ddia/ch03/code
uv run --with numpy --with scipy python3 sparse_matrix.py     # a few seconds

Do not read the output yet — make each prediction first.

Step 1 · the memory gap (the surprise)

Predict. A 10,000 × 5,000 matrix at 1% density — 50 million cells, ~500,000 actual ratings. How much memory does the dense array take, and how much does the sparse form take? What's the ratio?

How much memory for a 1%-full matrix, dense vs sparse?

ratings matrix: 10,000 users x 5,000 movies = 50,000,000 cells density: 1.0% (497,458 ratings) -- like a real recommender (Netflix was ~1.2%) memory to hold the matrix: dense (float32, every cell) = 200.0 MB sparse (CSR: value + col + row ptr) = 4.0 MB -> sparse is 50x smaller

200 MB vs 4 MB — 50×. The dense array pays for all 50 million cells even though 99% are empty; CSR pays only for the ~500,000 ratings that exist.

Step 2 · the break-even (predict the rule)

Predict. Sparse clearly wins at 1% full. At what density does it stop winning — where does storing only the nonzeros cost as much as storing every cell?

At what density does sparse stop winning?

break-even: CSR stores ~2 numbers per rating, so it saves memory below ~50% density; here the saving is 1/(2 x 0.01) = 50x.

CSR stores two numbers per rating (the value and its column index) where dense stores one per cell, so sparse wins whenever fewer than ~50% of cells are filled. Below that its advantage is simply 1 / (2 × density) — which is why 1% density buys exactly 50×.

Step 3 · at real scale

Predict. The Netflix Prize matrix was ~480,500 users × 17,770 movies at ~1.18% density. How big would that be dense vs. sparse?

How big is the Netflix-scale matrix each way?

at Netflix scale (480,500 x 17,770, 1.18% dense), analytic: dense ~ 34 GB vs sparse ~ 0.8 GB (42x)

The dense matrix is 34 GB — it wouldn't fit in memory — while the sparse form is under 1 GB. At real recommender scale, sparse storage isn't an optimization; it's the difference between fitting in RAM and not.

What you should see

10,000 × 5,000 at 1%: dense 200 MB vs sparse 4 MB → ~50×.
Break-even density ~50%; the saving is 1 / (2 × density).
Netflix scale: dense ~34 GB vs sparse ~0.8 GB (~42×).

Why

A dense matrix reserves storage by shape: rows × columns cells, each a fixed-size number, whether or not anything was ever put there. That's ideal when most cells are meaningful, but a ratings matrix is the opposite — a user rates dozens of movies out of tens of thousands, so ~99% of cells are structural zeros carrying no information. Storing them is pure waste, and the waste grows with the size of the catalog you didn't rate.

A sparse format flips the accounting to pay by content: it stores each nonzero value once, plus enough bookkeeping to know where it lives. CSR keeps three arrays — the values, their column indices, and a pointer per row into those arrays — so its cost is ~2 numbers per rating and essentially nothing for the empty cells. That is why the ratio is governed entirely by density: dense / sparse ≈ 1 / (2 × density) for a float32 value with a 32-bit index. At 1% you save 50×; the same data at 50% saves nothing, because by then you're storing two numbers per cell instead of one. The transform to a matrix is essential for the linear algebra the book describes; the sparse representation is what makes that transform affordable.

The boundary — sparsity is a property of the data, and formats specialize Sparse storage helps only when the matrix actually is sparse; a dense matrix (an image, a fully-populated feature table) stored as CSR would be larger than dense, since you'd pay the index overhead on every cell. And the format matters for the operation: CSR is fast for row slices and matrix-vector products (what recommenders do), CSC for column slices, COO for building incrementally — which is why libraries offer several and convert between them. The book's point stands: a matrix is the right model for recommendation, but at real scale you never materialize it densely — you keep it sparse and let the linear algebra run on the nonzeros.

Go deeper

Change the density. Set DENSITY to 0.001 (0.1%) and to 0.1 (10%), predicting each ratio from 1 / (2 × density) before running. Confirm the rule.
float64 values. Switch the dense array and CSR values to float64 and predict how the ratio changes (the index stays 32-bit, so the per-nonzero cost isn't simply doubled).
Cross the break-even. Push DENSITY to 0.5 and then 0.7. Predict where CSR stops saving memory and starts costing more than dense — and check that it matches the ~50% rule.

Sources: DDIA 2e, Ch. 3, §"DataFrames, Matrices, and Arrays" · SciPy scipy.sparse (CSR) · the Netflix Prize dataset dimensions.