# DDIA Ch. 3: the ratings matrix that's 99% empty ## Concept The book turns a table of movie ratings into a users × movies matrix (Figure 3-9) so linear algebra can drive recommendations. But real ratings matrices are almost entirely empty — a user rates a tiny fraction of all movies — so storing a number for *every* cell wastes nearly everything. You will hold a realistic 1%-full ratings matrix two ways and watch the dense array take **200 MB** while the sparse representation takes **4 MB** — **50× smaller** — with a clean rule for exactly when sparse wins. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 3 — *Data Models and Query Languages*, §"DataFrames, Matrices, and Arrays": > the data in Figure 3-9 could be a part of a system for recommending movies … > Once the data is in the form of a matrix of numbers, it is amenable to linear > algebra operations, which form the basis of many ML algorithms. The book shows the ratings-to-matrix transform. Here you measure the storage consequence the transform runs into — and the sparse representation that answers it. ## Prerequisites The surprise is about representing a mostly-empty matrix. These help; each line notes where it shows up. 1. **Dense vs. sparse storage** — dense stores a value for every cell; sparse stores only the nonzero entries plus where they are. 2. **Density (fill fraction)** — the share of cells that are nonzero. Recommender matrices are ~1% or less. 3. **CSR (compressed sparse row)** — a standard sparse format: three arrays — the nonzero values, their column indices, and per-row start pointers. 4. **Cost per entry** — dense costs one number per *cell*; CSR costs ~two numbers per *nonzero* (a value and a column index). That ratio sets the break-even. ### Where to learn them - **Sparse formats (#1, #3):** the SciPy `scipy.sparse` docs (CSR/CSC/COO); any numerical-computing text on sparse matrices. - **Why it matters for ML (#2):** DDIA §"DataFrames, Matrices, and Arrays" and the recommender-systems literature (the Netflix Prize matrix is the canonical example). If only one is new, make it #4 — the two-numbers-per-nonzero cost is the whole break-even. ## Environment Deterministic seed, so the sizes reproduce exactly: - macOS 26.5.2 (Darwin 25.5.0), arm64 - NumPy + SciPy on Python 3.12.13, run via `uv` — no Docker - A 10,000 × 5,000 ratings matrix at 1% density ## Mental model A ratings matrix: rows are users, columns are movies, a cell holds a star rating if that user rated that movie and is empty otherwise. We build a 10,000 × 5,000 matrix (50,000,000 cells) at **1% density** — like a real recommender (Netflix's was ~1.2%) — and store it two ways: - **Dense** — a `float32` NumPy array with a number in every one of the 50M cells, rated or not. - **Sparse (CSR)** — only the ~500,000 actual ratings, each as a value plus a column index, with per-row pointers. We compare `nbytes`. The whole exercise is one script, `code/sparse_matrix.py`. ## Setup ``` cd study/ddia/ch03/code uv run --with numpy --with scipy python3 sparse_matrix.py # a few seconds ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — the memory gap (the surprise) **Predict.** A 10,000 × 5,000 matrix at 1% density — 50 million cells, ~500,000 actual ratings. How much memory does the dense array take, and how much does the sparse form take? What's the ratio? **Observe.** ``` ratings matrix: 10,000 users x 5,000 movies = 50,000,000 cells density: 1.0% (497,458 ratings) -- like a real recommender (Netflix was ~1.2%) memory to hold the matrix: dense (float32, every cell) = 200.0 MB sparse (CSR: value + col + row ptr) = 4.0 MB -> sparse is 50x smaller ``` **200 MB vs 4 MB — 50×.** The dense array pays for all 50 million cells even though 99% are empty; CSR pays only for the ~500,000 ratings that exist. ### Step 2 — the break-even (predict the rule) **Predict.** Sparse clearly wins at 1% full. At what density does it *stop* winning — where does storing only the nonzeros cost as much as storing every cell? **Observe.** ``` break-even: CSR stores ~2 numbers per rating, so it saves memory below ~50% density; here the saving is 1/(2 x 0.01) = 50x. ``` CSR stores two numbers per rating (the value *and* its column index) where dense stores one per cell, so sparse wins whenever fewer than **~50%** of cells are filled. Below that its advantage is simply `1 / (2 × density)` — which is why 1% density buys exactly 50×. ### Step 3 — at real scale **Predict.** The Netflix Prize matrix was ~480,500 users × 17,770 movies at ~1.18% density. How big would that be dense vs. sparse? **Observe.** ``` at Netflix scale (480,500 x 17,770, 1.18% dense), analytic: dense ~ 34 GB vs sparse ~ 0.8 GB (42x) ``` The dense matrix is **34 GB** — it wouldn't fit in memory — while the sparse form is **under 1 GB**. At real recommender scale, sparse storage isn't an optimization; it's the difference between fitting in RAM and not. ## What you should see - 10,000 × 5,000 at 1%: dense **200 MB** vs sparse **4 MB** → **~50×**. - Break-even density ~**50%**; the saving is `1 / (2 × density)`. - Netflix scale: dense ~**34 GB** vs sparse ~**0.8 GB** (~**42×**). ## Why A dense matrix reserves storage by *shape*: rows × columns cells, each a fixed-size number, whether or not anything was ever put there. That's ideal when most cells are meaningful, but a ratings matrix is the opposite — a user rates dozens of movies out of tens of thousands, so ~99% of cells are structural zeros carrying no information. Storing them is pure waste, and the waste grows with the size of the catalog you *didn't* rate. A sparse format flips the accounting to pay by *content*: it stores each nonzero value once, plus enough bookkeeping to know where it lives. CSR keeps three arrays — the values, their column indices, and a pointer per row into those arrays — so its cost is ~2 numbers per rating and essentially nothing for the empty cells. That is why the ratio is governed entirely by density: `dense / sparse ≈ 1 / (2 × density)` for a float32 value with a 32-bit index. At 1% you save 50×; the same data at 50% saves nothing, because by then you're storing two numbers per cell instead of one. The transform to a matrix is essential for the linear algebra the book describes; the sparse representation is what makes that transform affordable. **The boundary — sparsity is a property of the data, and formats specialize.** Sparse storage helps only when the matrix actually *is* sparse; a dense matrix (an image, a fully-populated feature table) stored as CSR would be *larger* than dense, since you'd pay the index overhead on every cell. And the format matters for the operation: CSR is fast for row slices and matrix-vector products (what recommenders do), CSC for column slices, COO for building incrementally — which is why libraries offer several and convert between them. The book's point stands: a matrix is the right *model* for recommendation, but at real scale you never materialize it densely — you keep it sparse and let the linear algebra run on the nonzeros. ### Go deeper 1. **Change the density.** Set `DENSITY` to 0.001 (0.1%) and to 0.1 (10%), predicting each ratio from `1 / (2 × density)` before running. Confirm the rule. 2. **`float64` values.** Switch the dense array and CSR values to `float64` and predict how the ratio changes (the index stays 32-bit, so the per-nonzero cost isn't simply doubled). 3. **Cross the break-even.** Push `DENSITY` to 0.5 and then 0.7. Predict where CSR stops saving memory and starts costing more than dense — and check that it matches the ~50% rule.