The frontier map — territory 5 of 5
Information Theory & the Geometry of Learning
Bits as the currency of learning. Entropy, KL divergence and mutual information are where compression, generation and alignment turn out to be one subject — the territory that derives objectives (ELBO, InfoNCE, DPO) instead of guessing them.
The central picture: KL divergence is not symmetric — and that asymmetry is a design tool
Forward vs reverse KL — two ways to approximate a two-peaked truth
P — the true distribution (two peaks)
Q — the model (one peak, dashed)
The roadmap of understanding — three levels deep
Level 1 — Core objects
surprise, bits, and shared informationThe ideas
- Entropy: the irreducible surprise of a source
- Cross-entropy: surprise under the wrong model
- KL: the gap between them — extra bits paid
- Mutual information: what knowing X tells you about Y
Where it lives
- The LLM training objective, read as coding theory
- Every "distance between distributions" argument
What it unlocks
- Seeing prediction = compression: a better LLM is literally a better compressor of text
Level 2 — Working theory
bounds you can train onThe ideas
- ELBO: when the true likelihood is intractable, optimize a tractable lower bound
- InfoNCE: a trainable lower bound on mutual information — contrastive learning, derived
- Rate–distortion: the best possible compression at a given quality
- Forward vs reverse KL as a behavioral dial (the figure above)
Where it lives
- VAEs — and diffusion models, whose training loss is an ELBO in disguise
- CLIP and every contrastive embedder
- Distillation objectives (which KL, which temperature)
What it unlocks
- Deriving working objectives for intractable problems — the field's most reused trick
- Explaining why contrastive pretraining produces useful geometry
Level 3 — The frontier
geometry, bottlenecks, and alignmentThe ideas
- Fisher information: the local metric on the space of distributions — natural gradient follows it, not raw parameters
- Information bottleneck: representation = compress the input, keep what predicts the target
- Alignment as KL-regularized RL: maximize reward while staying close to the base model
- DPO: solve that objective in closed form — no reward model, no RL loop
Where it lives
- Natural-gradient lineage: K-FAC, TRPO's KL constraint
- The RLHF pipeline of every aligned chat model
- The DPO paper — a masterclass of this territory
What it unlocks
- Deriving alignment methods rather than memorizing them
- Reading "the KL term" in any paper and knowing which behavior it buys
- A principled account of what representations should keep and discard
Cheat sheet: concept → landmark → what it's doing there
| Concept | Landmark use | What it's doing there |
|---|---|---|
| Cross-entropy as coding | LLM pretraining | Prediction = compression; loss = bits per token |
| ELBO | VAE; diffusion training objectives | A trainable bound when the likelihood is intractable |
| InfoNCE | CLIP, contrastive embedders | A trainable bound on mutual information |
| KL direction | VAEs, distillation, variational inference | Mode-covering vs mode-seeking behavior, by choice |
| Fisher / natural gradient | K-FAC, TRPO | Steps measured in distribution space, not parameter space |
| KL-regularized RL | RLHF; DPO's closed form | Improve on preferences without leaving the base model |
| Information bottleneck | Representation-learning theory | What a good representation keeps, formally |
The territory's signature move: when the true objective is intractable, find a bound
you can train on — ELBO, InfoNCE and DPO are the same maneuver executed three times, a decade apart.
End of the map. Five territories, one loop back: the architect chooses what to build,
the engineer trains it, and this track supplies the mathematics both of them are standing on.
Back to the frontier overview · all field guides.