Pillar 1 · Model Selection — family deep dive

Types of Transformer LLMs

"LLM" is not one thing. Under the Transformer umbrella there are three architectural shapes and several commercial flavors — and they differ in what they can do, what they cost, and where they can run. As a buyer, the type decides the shortlist before any benchmark does.

The three architectural shapes

Every Transformer is one of these — the shape decides whether it understands, generates, or transforms.

Encoder-only vs decoder-only vs encoder–decoder

Rule of thumb: understand → encoder-only (cheap, runs on CPU) · generate → decoder-only (the chatbots) · transform → encoder–decoder (translation, speech-to-text).

Base vs instruct: every model above starts as a base model that merely continues text. The instruct / chat variant is additionally tuned to follow instructions — and it's what every API serves you. If a vendor benchmark quotes a base model, ask for instruct numbers.

The types you'll actually choose between

Decoder-only frontier models

the default "LLM" everyone means

prompt (text ± images) generated text · code · JSON

Named models

GPT-4o, GPT-4.1 (OpenAI)
Claude Sonnet / Opus (Anthropic)
Gemini Pro (Google)
Llama 3/4, Mistral Large, Qwen, DeepSeek-V3 (open-weight)

Choose it when

Chat, agents, code, extraction — the general case
Quality matters more than unit cost
You want one model covering many tasks

Watch out

Priciest per token; latency grows with output
Hosted APIs = data-processing agreements to review
Open-weight versions need serious GPUs (pillar 2)

Small language models (SLMs)

the cost & privacy play

prompt generated text — on your own hardware

Named models

Phi-4 (Microsoft)
Gemma 2/3 (Google)
Llama 3.2 1B–8B (Meta)
Mistral 7B / Ministral, Qwen 2.5 small

Choose it when

Data cannot leave your infrastructure
High volume where per-token API cost explodes
Edge / on-device deployment
As the target of fine-tuning for one narrow task

Watch out

Noticeably weaker reasoning — always pilot first
You own the serving stack, upgrades and evals
A fine-tuned SLM beats a prompted SLM, not a frontier model

Reasoning models

buy thinking time by the token

hard problem long internal reasoning answer

Named models

OpenAI o1 / o3
DeepSeek-R1
Claude with extended thinking
Gemini thinking variants

Choose it when

Multi-step analysis: math, planning, complex code
Agent workflows that must not derail
Accuracy on hard tasks beats response time

Watch out

Slower and much pricier — reasoning tokens bill too
Overkill for lookup, chat and simple extraction
Route: cheap model first, escalate hard cases

Mixture-of-Experts (MoE)

big brain, partial activation

prompt routed through a few "experts" per token text

Named models

Mixtral 8×7B / 8×22B (Mistral)
DeepSeek-V3 / R1
Qwen-MoE; several frontier APIs are MoE inside

Choose it when

Self-hosting: near-frontier quality at lower compute per token
High-throughput serving economics

Watch out

VRAM must hold all experts — memory like the big model, compute like a small one
Behind an API, MoE is invisible to you — it's a self-hosting concern

Encoder-only (BERT family)

the cheap understanding workhorse

text label · entities · score · vector (no generation)

Named models

BERT, RoBERTa, DeBERTa
DistilBERT / MiniLM (distilled, faster)
ModernBERT (2024 refresh)

Choose it when

Classification at scale: routing tickets, sentiment, PII detection
Named-entity extraction with stable schemas
Millisecond latency on CPU — no GPU bill

Watch out

Cannot generate text at all
Needs a fine-tuning pass per task (small labeled set)
Compare against "just prompt a cheap LLM" on total cost

Encoder–decoder (seq2seq)

input in, transformed output out

source sequence target sequence

Named models

T5 / FLAN-T5, BART (text transforms)
NLLB, MarianMT (translation)
Whisper (speech → text — same shape, audio in)

Choose it when

Dedicated translation pipelines at volume
Speech-to-text (Whisper is the default buy)
Fixed transform tasks where a small tuned model is cheapest

Watch out

Less general than decoder-only chat models
For low volumes, a frontier LLM already translates well

Multimodal LLMs (VLMs)

eyes on top of the language brain

text + images (± audio, video) text

Named models

GPT-4o (vision + audio)
Claude vision, Gemini (long video)
Llama 3.2 Vision, Qwen-VL (open-weight)

Choose it when

Documents where layout matters: invoices, forms, tables
Screenshot / UI understanding, visual QA
One API call instead of an OCR + LLM pipeline

Watch out

Images consume many input tokens — cost adds up
For exact character accuracy, dedicated OCR still wins

Cheat sheet: LLM type → when to buy it

Type	Named models	Buy it for	Ops note
Decoder-only frontier	GPT-4o, Claude, Gemini, Llama 3/4	General chat, agents, code — the default	Highest per-token cost; API or big GPUs
Small language model	Phi-4, Gemma, Llama 3.2 8B, Mistral 7B	Privacy, volume economics, edge	Fits one 24 GB GPU (see pillar 2)
Reasoning model	o3, DeepSeek-R1, Claude extended thinking	Hard multi-step problems only	Slow; reasoning tokens bill as output
Mixture-of-Experts	Mixtral, DeepSeek-V3	Self-hosted quality per compute dollar	VRAM sized to all experts
Encoder-only	BERT, DeBERTa, DistilBERT	Classification / NER at scale	CPU-friendly, milliseconds, tiny
Encoder–decoder	T5, NLLB, Whisper	Translation, speech-to-text	Small, cheap, task-fixed
Multimodal (VLM)	GPT-4o, Gemini, Qwen-VL	Documents with layout, screenshots	Image tokens inflate input cost

Portfolio thinking: real architectures route — an encoder-only classifier triages, a small model handles the routine 80%, and a frontier or reasoning model gets the expensive 20%. Choosing "one LLM for everything" is how token bills surprise CFOs.