The frontier map — territory 3 of 5

Statistical Learning Theory

The deepest question in the field: why does minimizing training loss produce models that work on data they've never seen? Classical theory built elegant answers; deep learning broke them; and the repairs — double descent, implicit bias, scaling laws — are where this territory's frontier runs today.

The central picture: the curve that broke classical theory

Double descent — test error vs model capacity

Classical theory predicted only the left half. The right half — where all modern models live — is why this territory is an open frontier rather than a settled textbook.

The roadmap of understanding — three levels deep

Level 1 — Core objects

the vocabulary of generalization

train vs test · overfitting · bias–variance · capacity

The ideas

Generalization gap: train performance − test performance
Bias–variance: too simple misses structure, too flexible memorizes noise
Capacity: how rich the model's hypothesis space is

Where it lives

Every validation curve ever plotted
Regularization decisions across all of ML

What it unlocks

Framing any "does it really work?" question precisely

Level 2 — Working theory

the classical machinery

uniform convergence · VC dimension · Rademacher complexity · PAC bounds

The ideas

Concentration inequalities: averages behave predictably
Uniform convergence: the whole hypothesis class behaves at once
VC / Rademacher: measuring a class's effective richness
PAC framing: "probably approximately correct" guarantees

Where it lives

Guarantees for classic ML (SVMs, boosting had beautiful theory)
The baseline against which deep learning's behavior is measured

What it unlocks

Writing and judging generalization claims rigorously
Understanding exactly what deep learning violated

Level 3 — The frontier

the modern puzzles & the laws that pay the bills

double descent · implicit bias of SGD · scaling laws · grokking

The ideas

Benign overfitting: interpolating noise yet generalizing anyway
Implicit bias: SGD prefers simple solutions without being told to
Neural scaling laws: loss falls as a power law in params/data/compute
Compute-optimal training (Chinchilla): ~20 tokens per parameter
Grokking: generalization arriving long after memorization

Where it lives

Kaplan & Chinchilla scaling-law papers — the planning documents of every LLM lab
Belkin's double-descent line of work
Emergent-abilities and grokking literature

What it unlocks

Predicting what a 10× compute budget will buy before spending it
Choosing model size vs dataset size on paper
Judging "emergence" claims with the right skepticism

Cheat sheet: concept → landmark → what it's doing there

Concept	Landmark use	What it's doing there
Bias–variance	All classical model selection	The original story of under/overfitting
VC / Rademacher	SVM & boosting theory	Capacity measures with real guarantees
Double descent	Belkin et al.	The curve that reopened the field
Implicit bias of SGD	Modern generalization theory	Why overparameterized nets pick good solutions
Scaling laws	Kaplan et al., Chinchilla	Loss as a predictable function of scale — LLM planning math
Grokking	Small-model phenomenology	A lab for studying when understanding emerges

The territory's signature move: treat "it works" as a theorem to be earned. The scaling-law papers are the purest example — fit the law on small runs, then bet hundreds of millions of dollars of compute on the extrapolation.