The frontier map — territory 1 of 5

Analysis & Matrix Calculus

Derivatives grown up. Where the engineer trusts autograd, the scientist works with the objects autograd is made of — Jacobians, Hessians, curvature, smoothness — and uses them to derive new layers, predict training dynamics and design optimizers.

The central picture: backprop is a product of Jacobians

Chain rule, in its full matrix form

The engineer's loss.backward() is this equation. The scientist can write it down, reorder it (forward vs reverse mode), and exploit its structure (checkpointing, custom VJPs).

The roadmap of understanding — three levels deep

Level 1 — Core objects

the vocabulary of multivariable change

gradient · Jacobian · Hessian · Taylor expansion

The ideas

Gradient: sensitivity vector for scalar outputs
Jacobian: the full sensitivity matrix for vector outputs
Hessian: curvature — how the gradient itself changes
Taylor expansion: every local analysis starts here

Where it lives

Method sections of every deep-learning paper
The definition of autodiff itself

What it unlocks

Reading ∂-notation as fluently as code
Seeing "linearize and analyze" as a universal move

Level 2 — Working theory

matrix calculus as a craft

matrix differentiation rules · vector-Jacobian products · conditioning · Lipschitz smoothness

The ideas

The identities: d(Ax) = A, d(xᵀAx) = (A+Aᵀ)x, and friends
VJPs / JVPs: reverse vs forward mode, and when each wins
Condition number: how lopsided the curvature is
Smoothness (Lipschitz constants): how far a gradient step can be trusted

Where it lives

Deriving the backward pass of a novel layer (custom autograd functions)
Choosing learning rates from smoothness (η < 2/L)
Spectral normalization in GAN discriminators

What it unlocks

Inventing layers autograd has never seen (and getting their gradients right)
Explaining why normalization makes optimization easier: it repairs conditioning

Level 3 — The frontier

training dynamics as solvable equations

NTK · sharpness & flat minima · second-order methods · implicit differentiation

The ideas

Neural Tangent Kernel: infinitely wide nets train as linear models — dynamics in closed form
Sharpness: Hessian eigenvalues at the minimum predict generalization (→ SAM optimizer)
Second-order & preconditioned methods: K-FAC, Shampoo, Muon
Implicit differentiation: gradients through "solve this subproblem" (meta-learning, DEQs)

Where it lives

NTK papers and their infinite-width analyses
The SAM line of optimizers (seek flat minima explicitly)
Modern optimizer research (Shampoo/Muon at LLM scale)

What it unlocks

Predicting training behavior before running it
Designing optimizers, not just configuring them
The analysis half of every "why does this trick work?" paper

Cheat sheet: concept → landmark → what it's doing there

Concept	Landmark use	What it's doing there
Jacobian products	Backpropagation / autodiff	All gradients in one reverse sweep — the field's founding theorem
Hessian eigenvalues	SAM, curvature analyses	Sharp vs flat minima — a generalization signal
Condition number	Normalization layers, preconditioning	Why some landscapes are slow and what repairs them
Lipschitz constants	Spectral norm in GANs; step-size theory	Certifying how far a step can be trusted
NTK	Infinite-width theory	A solvable model of deep-network training
Implicit differentiation	Meta-learning, hyperparameter optimization, DEQs	Differentiating through an inner optimization

The territory's signature move: linearize, then analyze. Nearly every theory result in deep learning — from NTK to sharpness to step-size rules — is a Taylor expansion taken seriously.