The 50% map — part 4 of 5

Losses & Information Theory — the goal is the product

Whatever the loss rewards, the model becomes. The architect's guide showed that model families differ by their loss — a CNN scores classification error, a GAN plays minimax, a Transformer predicts tokens. This page is the same map seen from the inside: what each objective says, and what behavior it buys.

The heart of it: cross-entropy

One curve explains classifier training, LLM pretraining and the perplexity metric at once.

Loss = −log p(correct answer)

LLM pretraining is this curve applied to the next token, trillions of times. Perplexity — the LLM report card — is just e^{average loss}: "how many tokens was the model effectively choosing between?"

The roadmap of understanding — three levels deep

Level 1 — Intuition

loss, surprise, and distance between beliefs

loss = one number scoring predictions · entropy = average surprise · KL = extra surprise

The ideas

A loss compresses "how wrong were we" into one number to minimize
Entropy: how surprising a source is on average
Cross-entropy: your surprise when using the wrong beliefs
KL divergence: the gap between two distributions

Where it lives

The number on every training dashboard
The objective line of every paper's method section

What it explains

Why "what does the loss reward?" is the first question about any model's behavior
Why prediction and compression are the same skill

Level 2 — Working fluency

the standard objectives

MSE · cross-entropy · perplexity · class weighting

The ideas

MSE: match numeric targets — regression's default
Cross-entropy: put probability mass on the right class/token
Perplexity = e^{cross-entropy} — surprise per token
Label smoothing & class weights: reshape what "right" pays

Where it lives

LLM pretraining: next-token cross-entropy, nothing more exotic
Model cards: "perplexity 8 on held-out data"
Fraud/defect training: weighting the rare class up

What it explains

What "loss 2.1 → perplexity ≈ 8" actually says
Why accuracy can rise while loss worsens (confidence shifting)
Why imbalanced data quietly trains a useless model without reweighting

Level 3 — The deep end

shaped objectives — one per model family

contrastive · minimax & Wasserstein · diffusion · KL & preference losses

The ideas

Contrastive (InfoNCE): pull matching pairs together, push others apart
Minimax: generator vs discriminator, loss as a game
Wasserstein distance: a better-behaved gap between distributions
Diffusion objective: predict the added noise (a weighted MSE)
KL as a leash: stay close to a reference model
Preference losses (RLHF reward, DPO): prefer the chosen answer

Where it lives

CLIP & every embedding model → the geometry RAG relies on
GANs; WGAN when vanilla training collapses
Stable Diffusion's inner training loop
Alignment fine-tuning of every chat assistant

What it explains

Why each model family behaves the way it does — the loss is the family trait
The famous fix: a GAN's vanishing gradients cured by switching to Wasserstein loss — changing the geometry, not the network
Why chat models feel "aligned": a KL leash to the base model plus preference pressure

Cheat sheet: objective → who uses it → what it teaches

Objective	Used by	What it teaches the model
MSE	Regression; diffusion's inner loop	Match the numeric target (or the exact noise)
Cross-entropy	Classifiers; LLM pretraining	Put probability mass on the correct class/token
Perplexity	LLM evaluation (a metric, not a loss)	Reports surprise per token — lower is better
Contrastive / InfoNCE	CLIP, text embedders	A geometry where similar things sit close
Minimax / Wasserstein	GANs / WGAN	Realism via competition; W-distance stabilizes it
KL divergence	Distillation, RLHF penalty, VAE	Don't stray far from the reference distribution
Preference losses (DPO, RLHF)	Alignment fine-tuning	Prefer answers humans chose over ones they rejected

The load-bearing insight: to predict a model's behavior — including its failure modes — ask what its loss paid for. Embedders cluster because InfoNCE paid for clustering; chat models flatter because preference data paid for approval; LLMs guess confidently because cross-entropy never paid for saying "I don't know".