Pillar 2 of 3
Model Operational Metrics
This is the math you do need — and it's operational, not algorithmic. Four numbers drive almost every architectural decision: how much a model can read at once, how fast it responds, how much hardware it needs, and how good its answers actually are.
Context window — how much can it process at once?
An LLM can only "see" what fits in its context window, measured in tokens. Everything counts against it: system instructions, retrieved documents, conversation history, and the answer being generated. The window is your hard budget when designing prompts and RAG.
1 token ≈ ¾ word
1,000 tokens ≈ 750 English words ≈ 1½ pages. API pricing is per token — input and output both bill.
4K–8K window
A few pages. Enough for a chat turn plus a short document. Typical of small local models.
128K–200K window
A whole book or codebase slice. Frontier hosted models. Lets RAG pass in many retrieved chunks.
Bigger ≠ better recall
Models attend unevenly across huge contexts ("lost in the middle") — and every token costs money and latency. Retrieve selectively instead of dumping everything in.
Latency vs throughput — which one does your use case buy?
Latency is how long one user waits for one answer. Throughput is how much total work the system completes per unit of time. GPUs love batching, so the two trade off: tuning a deployment for throughput (big batches) makes individual answers slower, and vice versa.
| Scenario | Optimize for | Feels right when | Architecture lever |
|---|---|---|---|
| Chat assistant | Latency | First token < 1 s, then 30–100 tokens/s streamed | Streaming responses, small/fast model for first draft |
| Document pipeline (overnight batch) | Throughput | Documents/hour per GPU, not seconds/doc | Batching, queues, spot/off-peak compute |
| Image generation | Latency (perceived) | 2–10 s with progress feedback | Fewer diffusion steps, smaller resolution previews |
| Real-time vision (defect detection) | Both — hard limits | Every frame in < 30–50 ms | Small CNN on edge hardware next to the camera |
Compute & VRAM — will it fit on the GPU?
The sizing rule of thumb: at FP16 precision a model needs about 2 GB of VRAM per 1 billion parameters just to load the weights. Quantization stores weights in fewer bits — INT8 halves the footprint, 4-bit quarters it — with a modest quality cost that is often acceptable in production. Then add ~10–20% headroom for the KV cache and activations.
VRAM needed to load model weights, by size and precision
GB of GPU memory · weights only — add ~10–20% for KV cache & activations
Evaluation metrics — reading a Data Scientist's report
You won't compute these, but you must judge production readiness from them. For classifiers, everything starts from four possible outcomes of a prediction:
The confusion matrix behind precision & recall
| Metric | Plain-language question | Prioritize it when… |
|---|---|---|
| Precision | "When the model flags something, how often is it right?" (TP / TP+FP) | False alarms are expensive — fraud alerts, content takedowns, anything that pages a human |
| Recall | "Of all the real cases, how many did it find?" (TP / TP+FN) | Missing a case is expensive — medical screening, safety defects, compliance checks |
| F1-score | "Balanced single number combining both" | You need one comparable score, and both error types matter roughly equally |
| FID | "How close do generated images look to real ones?" (lower = better) | Judging generative image models (GANs, diffusion) — pair it with human review |