The 50% map — part 2 of 5

Calculus & Optimization — how models learn

"Training" is one idea repeated billions of times: measure how the loss reacts to each weight (the gradient), then nudge every weight downhill (gradient descent). The chain rule makes this computable for a billion parameters at once — that's backpropagation, and everything in this part orbits it.

Where it lives: the backward pass

Backprop = the chain rule walked backwards over the network

The chain rule is why deep learning is possible at all: one forward + one backward pass prices the influence of every parameter on the loss.

The roadmap of understanding — three levels deep

Level 1 — Intuition

what you must be able to picture

derivative = sensitivity · gradient = steepest direction · learning = walking downhill

The ideas

A derivative answers: "if I nudge this, how much does that move?"
The gradient collects those answers for all weights at once
The loss is a landscape; training walks it downhill

Where it lives

The loss curve in every training dashboard
loss.backward() — the sensitivity of L to everything

What it explains

What "training" literally does, step by step
Why the loss curve is the heartbeat monitor of a run

Level 2 — Working fluency

the optimizer config, understood

w ← w − η·∇L · SGD & mini-batches · Adam · learning-rate schedules

The ideas

The update rule: step size η (learning rate) × direction ∇L
Mini-batch SGD: noisy but cheap gradient estimates
Momentum: keep rolling through small bumps
Adam/AdamW: a per-weight adaptive step size — the default
Schedules: warmup (gentle start), cosine decay (fine finish)
Weight decay: steady pull toward smaller weights

Where it lives

Every optimizer config block in every training script
LLM training recipes: AdamW + warmup + cosine, almost always
Fine-tuning: small LR (don't stomp on pretrained weights)

What it explains

Why LR is the single most important hyperparameter
Loss spiking at start → warmup missing or LR too high
Why "same run, different batch size" needs a different LR

Level 3 — The deep end

gradient flow, and architecture as its medicine

vanishing / exploding gradients · loss landscapes · why residuals & norms exist

The ideas

Deep chains multiply sensitivities — products of small numbers vanish, of large ones explode
Loss landscapes: plateaus, cliffs, saddle points
Adam as a rough "per-direction ruler" for uneven landscapes

Where it lives

Residual connections: a gradient highway past every layer
LayerNorm/RMSNorm: keep activations (and gradients) in range
ReLU-family activations: don't crush gradients like sigmoid did
Gradient clipping: hard cap on explosion

What it explains

Why modern architectures look the way they do — architecture is frozen optimization advice
Why GANs suffered vanishing gradients — and why changing the loss (WGAN) fixed the geometry of descent
Your NaN checklist: LR, clipping, norms, data

Cheat sheet: concept → where → purpose

Concept	Where you meet it	What it's doing there
Chain rule	`loss.backward()` — autograd	All gradients for a billion weights in one pass
Gradient descent	`optimizer.step()`	The actual learning: every weight nudged downhill
Learning rate	The first number in every config	Step size — too big diverges, too small crawls
Adam / AdamW	Default optimizer nearly everywhere	Per-weight adaptive steps for uneven landscapes
Warmup + cosine schedule	LLM and fine-tuning recipes	Stability early, precision late
Gradient clipping	Transformer & RNN training loops	Insurance against exploding updates
Residuals + normalization	Every modern block design	Keeping gradients alive through 100+ layers

The load-bearing insight: when training misbehaves, think in gradients, not in code. Divergence, plateaus, dead layers and NaNs are all statements about gradient flow — and the fixes (clipping, warmup, norms, residuals, a different loss) are all gradient-flow repairs.