The 50% map — part 1 of 5
Linear Algebra — the language models are written in
Every tensor, layer, embedding and attention score is linear algebra. Roughly 90% of what AI hardware computes is matrix multiplication. Master this part and model code stops being mysterious — it becomes bookkeeping over shapes.
Where it lives: one transformer layer, annotated
The scariest diagram in AI is four matmuls and a softmax.
A transformer layer as shape algebra
The roadmap of understanding — three levels deep
Level 1 — Intuition
what you must be able to pictureThe ideas
- A vector is a point in space — and a row of data, and a "meaning"
- A matrix transforms space (rotate, stretch, project)
- Matrix multiplication chains transformations
- Dot product measures alignment — similarity
Where it lives
- Embeddings: "meaning of a word" = a 1,000-dim vector
- Vector search / RAG: nearest points = related content
- Every dense layer: transform the representation
What it explains
- Why "distance = meaning" works at all
- Why a model is literally a stack of transformations
- Why similar inputs produce similar outputs
Level 2 — Working fluency
the daily language of model codeThe ideas
- Shape rule: (m×n)·(n×p) → (m×p) — inner dims must match
- Tensors = stacked matrices: batch × sequence × hidden
- Transpose, broadcasting, element-wise vs matrix ops
- Norms (L1/L2) — the "size" of a vector or gradient
- Cosine similarity = normalized dot product
Where it lives
- Attention: softmax(QKᵀ/√D)·V — read it now, shape by shape
- Gradient clipping: cap the L2 norm of the update
- Weight decay: penalize large weight norms
- Every framework error message you'll ever see
What it explains
- The #1 daily bug class: shape mismatches
- Why attention cost is quadratic in context length
- Where a model's parameter count comes from
Level 3 — The deep end
reach for it when you need itThe ideas
- Rank: how much independent information a matrix carries
- Low-rank approximation: big matrix ≈ product of two thin ones
- SVD: any matrix = rotate · stretch · rotate
- Eigenvalues: directions a transform only stretches
Where it lives
- LoRA: fine-tune ΔW = A·B with rank 8 instead of the full matrix
- PCA: project embeddings to 2-D for visualization
- Model compression & distillation of weight matrices
- Initialization schemes and rotary position embeddings
What it explains
- Why LoRA with r=8 can steer a 7B model: weight updates are naturally low-rank
- Why PCA plots of embeddings show meaningful clusters
- Why bad initialization explodes activations
Cheat sheet: concept → where → purpose
| Concept | Where you meet it | What it's doing there |
|---|---|---|
| Dot product | Attention scores QKᵀ · cosine similarity in vector DBs | Measuring how related two things are |
| Matrix multiplication | Dense, conv and attention layers — all of them | Transforming representations; 90% of all FLOPs |
| Shapes (B, S, D) | Every line of PyTorch / JAX you'll read | The type system of ML — most bugs are shape bugs |
| Norms (L1 / L2) | Gradient clipping, weight decay, RMSNorm | Keeping magnitudes in a healthy range |
| Broadcasting | Bias adds, normalization layers, masking | One small tensor applied across a big one |
| Low-rank / SVD | LoRA adapters, compression | Cheap fine-tuning: two thin matrices instead of one huge one |
| Eigenvalues / PCA | Embedding visualization, spectral tricks, stability analysis | Finding the directions that matter in a learned space |
The one habit that carries the whole part: read model code shape-first. Annotate every
tensor with its (B, S, D) shape and every equation in a paper becomes checkable bookkeeping — including
attention, which is three matmuls and a softmax.