Architectural Traits of AI

Pillar 1 of 3

Model Capabilities & Selection Taxonomies

Don't learn what CNNs or GANs are inside — learn their input → output matchings. Every model family is a box with a characteristic shape: what goes in, what comes out, and what it's naturally good at. Match the requirement's shape to the box's shape and the choice is usually obvious.

The shape of the requirement picks the family

Start from the data and the task, not from the technology.

Requirement → model family, at a glance

What does the requirement look like? language / reasoning analyze images or video generate media predict from tables find similar things Transformer LLM GPT-4 · Claude · Llama — chat, summarize, extract CNN vision model YOLO · ResNet — detect, classify, segment Diffusion model (or GAN) Stable Diffusion · DALL·E · StyleGAN — images, audio Gradient boosting (classic ML) XGBoost · LightGBM — churn, fraud, scoring Embeddings + vector DB CLIP · text-embedding — search, RAG, clustering
Example from the field: "detect defects in factory pipes from video" → analyze images → CNN object detection (YOLO), not a GAN.
The one trait to memorize per family is its I/O shape. When a stakeholder describes a requirement, translate it into "what data goes in, what decision or artifact comes out" — then the family (and usually a short list of concrete models) falls out immediately.

The model families and their traits

For each: what it does, when to choose it, its constraints, and real deployments.

Transformer LLM

the universal text machine
text / tokens (± images) text · structured output · code

Predicts the next token using dot-product attention over everything in its context window. For you that means: one model family covers chat, summarization, extraction, translation, classification and code — steered by the prompt rather than by retraining.

Choose it when

  • Assistants & chatbots over company knowledge (with RAG)
  • Summarizing, extracting fields from documents
  • Classification with few or no training examples
  • Code generation and developer tooling

Watch out

  • Hallucinates confidently — ground facts with RAG
  • Context window caps how much it can read at once
  • Per-token cost and latency grow with volume
  • Data privacy terms of hosted APIs

Real examples

  • Customer-service assistant over product docs
  • Contract clause extraction for legal review
  • Call-center conversation summaries into CRM

Go deeper: the types of LLMs (GPT, Claude, BERT, T5, Mixtral…) →

CNN — Convolutional Neural Network

the workhorse of computer vision
images / video frames labels · bounding boxes · masks

Scans images with learned filters that detect edges, textures and shapes, building up to whole objects. Mature, fast, and small enough to run on edge devices and factory cameras. Comes in task-specific variants — classification, detection, segmentation — explained on the CNN deep-dive page.

Choose it when

  • Detecting, counting or classifying objects in images/video
  • Quality inspection and defect detection
  • Real-time or on-device inference (low power, low latency)

Watch out

  • Needs labeled examples of exactly what you detect
  • Accuracy drops on new conditions: lighting, angles, new product variants
  • Plan a retraining loop as the environment changes

Real examples

  • Defect detection in factory pipes from video (YOLO)
  • Shelf-stock recognition in retail
  • Medical imaging triage (X-ray screening)

Go deeper: CNN variants (ResNet, YOLO, U-Net, ViT…) →

Diffusion model

today's default for media generation
text prompt / noise images · audio · video

Starts from noise and removes it step by step until an image matching the prompt emerges. Produces the highest-quality, most controllable generations — at the price of running many denoising steps per output.

Choose it when

  • Marketing visuals, product mockups, concept art
  • Image editing: inpainting, style transfer, variants
  • Text-to-image where quality and prompt control matter

Watch out

  • Slow inference — seconds per image, GPU-hungry
  • Brand safety, IP and licensing of generated content
  • Prompt control is approximate; budget review loops

Real examples

  • Stable Diffusion product-shot variants for e-commerce
  • DALL·E ad-creative exploration
  • Architectural / interior design visualization

Go deeper: diffusion types (SDXL, FLUX, DALL·E, Sora…) →

GAN — Generative Adversarial Network

generates data via competition
random noise (± conditioning) realistic samples in one shot

Two networks compete: a generator forges samples, a discriminator calls out fakes, and both improve until forgeries look real. You don't need the game theory — you need to know it generates in a single forward pass, which makes it faster than diffusion at inference time.

Choose it when

  • Super-resolution / upscaling of images and video
  • Synthetic training data (incl. privacy-preserving tabular data)
  • Fast generation where diffusion's latency is too high

Watch out

  • Unstable to train → fewer reliable off-the-shelf options
  • Largely superseded by diffusion for text-to-image
  • Mode collapse: outputs can lack variety

Real examples

  • ESRGAN photo/video upscaling
  • StyleGAN avatar and face generation
  • Synthetic patient records for analytics without PII

Go deeper: GAN types (StyleGAN, CycleGAN, ESRGAN, CTGAN…) →

Gradient boosting & classic ML

still the champion on business tables
rows of features prediction · score · ranking

Ensembles of decision trees, each correcting the last. On structured, tabular business data these routinely beat deep learning — while being cheaper, faster (CPU-only inference), and far easier to explain to auditors and regulators.

Choose it when

  • Churn, fraud, credit scoring, pricing, propensity
  • You need explainability (feature importance) for compliance
  • Cheap, high-volume inference without GPUs

Watch out

  • Needs feature engineering and clean data pipelines
  • Doesn't consume raw text/images — pair with embeddings
  • Data drift: plan a retraining cadence

Real examples

  • XGBoost churn prediction from CRM data
  • LightGBM real-time fraud scoring
  • Search-result ranking in e-commerce

Go deeper: classic ML types (XGBoost, Prophet, k-means…) →

Embedding models

turn meaning into geometry
text · images dense vectors (similar meaning = nearby points)

Maps content into a vector space where distance means semantic similarity. Paired with a vector database, this is the retrieval backbone of RAG and of every "find things like this" feature.

Choose it when

  • Semantic search that understands synonyms and intent
  • The retrieval step of a RAG architecture
  • Deduplication, clustering, recommendations

Watch out

  • Switching embedding models means re-indexing everything
  • Heavy domain jargon may need a domain-tuned embedder
  • Similar ≠ correct — retrieval quality caps RAG quality

Real examples

  • "Find similar support tickets" for agents
  • Product search tolerant of typos and synonyms
  • Document retriever feeding an LLM assistant

Go deeper: embedding types (text-embedding-3, BGE, CLIP…) →

Go deeper: one page per family

Each family has types of its own — the real selection skill is picking the right variant, with named models for each.

Transformer LLMs

Decoder vs encoder vs encoder–decoder, Mixture-of-Experts, reasoning models, small vs frontier — GPT-4o, Claude, Llama, BERT, T5, Mixtral…

LLM types →

CNNs & vision models

"Feed-forward convolutional network" decoded, plus classification, detection, segmentation, ViT, OCR — ResNet, YOLO, U-Net, Mask R-CNN…

CNN types →

Diffusion models

Text-to-image, inpainting, ControlNet & LoRA, video and audio — Stable Diffusion, FLUX, DALL·E, Midjourney, Sora, Runway…

Diffusion types →

GANs

Conditional GANs, StyleGAN, image-to-image translation, super-resolution, synthetic tabular data — StyleGAN3, CycleGAN, ESRGAN, CTGAN…

GAN types →

Classic ML

Regression to gradient boosting, clustering, anomaly detection, forecasting — XGBoost, LightGBM, CatBoost, Prophet, Isolation Forest…

Classic ML types →

Embedding models

Text & multimodal embedders, bi-encoders vs rerankers, vector index ops — text-embedding-3, BGE, E5, CLIP, Cohere Rerank…

Embedding types →

Cheat sheet: requirement → model

When the requirement says…Reach forWhy
"Answer questions about our documents"LLM + RAG (embeddings + vector DB)Grounds answers in your data; no retraining needed
"Detect defects in factory video"CNN object detection (YOLO)Real-time boxes on frames; runs on edge hardware
"Generate product images / ad creatives"Diffusion (Stable Diffusion, DALL·E)Best quality & prompt control for text-to-image
"Upscale old footage" / "synthetic data"GAN (ESRGAN, StyleGAN)One-pass generation, mature for these niches
"Predict churn / fraud from our tables"Gradient boosting (XGBoost)Beats deep learning on tabular; explainable & cheap
"Search that understands meaning"Embeddings + vector DBDistance in vector space = semantic similarity
"Summarize every call / meeting"LLMText in, text out — its native shape
← OverviewArchitectural Traits of AI