The 50% map — part 3 of 5
Probability & Statistics — models as uncertainty machines
A model never "knows" anything — it scores possibilities. Every classifier output and every LLM token is a probability distribution, training means making the data probable, and evaluating models is statistics whether you treat it that way or not.
Where it lives: every token an LLM emits
Decoding is distribution surgery
The roadmap of understanding — three levels deep
Level 1 — Intuition
what you must be able to pictureThe ideas
- Probability quantifies uncertainty on a 0–1 scale
- A distribution lists outcomes and their weights
- Expectation = long-run average; variance = spread
- Conditioning: probabilities shift as evidence arrives
Where it lives
- Classifier outputs: "defect, p = 0.92"
- The next-token distribution behind every LLM word
What it explains
- Why models are never certain — only weighted
- Why the same prompt can yield different answers
Level 2 — Working fluency
distributions you'll actually touchThe ideas
- Bernoulli/categorical: the shape of classification and next-token prediction
- Gaussian: the shape of noise — init, diffusion, measurement error
- Softmax: scores → a categorical distribution
- MLE: training = maximize the probability of the data — minimizing −log p, which is cross-entropy
- Sampling: temperature sharpens/flattens; top-p trims the tail
Where it lives
- Every output layer ends in softmax or sigmoid
- Diffusion models literally add and remove Gaussian noise
- Generation configs: temperature, top_p, top_k
What it explains
- Where cross-entropy comes from (it isn't arbitrary)
- What temperature really does — and when 0 is right
- Hallucination as miscalibration: high confidence, wrong content
Level 3 — The deep end
the statistics of trusting a modelThe ideas
- Calibration: does p = 0.9 come true 90% of the time?
- Metrics are random variables — they have variance
- Confidence intervals & significance: is a win real or noise?
- Distribution shift: production data drifts away from training data
Where it lives
- Risk decisions built on model scores (fraud thresholds)
- Model comparisons and eval reports
- Monitoring dashboards, drift alerts, retraining triggers
What it explains
- When "+0.5% on the benchmark" means nothing
- Why accuracy quietly decays after deployment
- Why a threshold tuned in the lab misfires in production
Cheat sheet: concept → where → purpose
| Concept | Where you meet it | What it's doing there |
|---|---|---|
| Softmax + categorical | Every classifier & LLM output layer | Turning raw scores into a proper distribution |
| Temperature / top-p | Generation configs | Reshaping the distribution before the draw — determinism ↔ creativity |
| MLE / −log p | Cross-entropy training everywhere | "Make the observed data probable" |
| Gaussian | Weight init, diffusion noise schedules | Tractable, well-understood randomness |
| Calibration | Thresholding model scores in production | Whether p = 0.92 deserves your trust |
| Confidence intervals / A-B | Eval reports, model comparisons | Separating real improvements from noise |
| Drift statistics | Production monitoring | Detecting when the world moved and retraining is due |
The load-bearing insight: an LLM is a categorical distribution over ~100,000 tokens,
sampled once per token. Hold that picture and temperature, top-p, beam search, hallucination and
"why did it answer differently this time" all become one topic instead of five mysteries.