Pillar 1 · Model Selection — family deep dive

Types of Embedding Models

Embeddings turn meaning into geometry: similar content lands at nearby points in vector space. The types differ in what they embed (text, images, code) and how they compare (fast one-vector matching vs slow precise reranking). Together they form the retrieval layer under every RAG system and every "find things like this" feature.

The pattern that matters: retrieve fast, rerank precisely

A bi-encoder embeds query and documents separately — comparison is a cheap vector-distance lookup, so it scans millions of documents in milliseconds but only approximately. A cross-encoder (reranker) reads query and document together — far more accurate, far too slow for millions. Production search uses both, in stages:

Two-stage retrieval — the standard search & RAG pipeline

Query "refund policy?" Bi-encoder + vector index millions of docs · milliseconds top-50 Cross-encoder reranker precise · ~100 ms for 50 docs top-5 LLM / user grounded answer recall first (don't miss the right document), precision second (put it on top) skipping the reranker is the cheapest RAG quality upgrade teams leave on the table
Hybrid search adds classic keyword scoring (BM25) alongside vectors — exact part numbers and names still deserve exact matching.

The types, with named models

TypeNamed modelsChoose it whenWatch out
Text embeddings (bi-encoder)OpenAI text-embedding-3-small/-large, Cohere Embed v3, BGE-M3, E5, GTE, Voyage AISemantic search, RAG retrieval, dedup, clustering — the default buyModel change ⇒ re-embed the whole corpus; pin versions
Rerankers (cross-encoder)Cohere Rerank 3, bge-reranker, Jina Reranker, MiniLM cross-encodersQuality boost on top of any retriever — usually the best accuracy-per-dollar upgrade in RAGOnly rescores candidates — can't rescue a retriever that missed the document
Multimodal embeddingsCLIP, SigLIP, OpenCLIPText↔image search ("find photos of damaged pallets"), zero-shot image tagging, moderationWeaker on fine-grained text inside images — that's OCR's job
Late-interactionColBERT v2Middle ground: near cross-encoder quality at near bi-encoder speedBigger index; fewer managed offerings
Domain-tuned embeddingsvoyage-code (code), legal/medical fine-tunes, in-house tuned BGEHeavy jargon corpora where general embedders confuse termsMaintenance burden — benchmark against general models first
The infrastructure around them: vectors live in a vector database — pgvector (inside Postgres you already run), Qdrant, Weaviate, Milvus, Pinecone, or FAISS as a library — indexed with approximate-nearest-neighbor structures (HNSW) that trade a sliver of recall for millisecond lookups. Dimensions (384 → 3072) scale storage and speed linearly: half the dimensions ≈ half the cost.
Two lock-ins to plan for: (1) embeddings from different models don't mix — one model version per index, and a scheduled re-embed to upgrade; (2) retrieval quality caps RAG quality (pillar 3) — budget evaluation of the retriever itself, not just the LLM's answers.
← Classic ML typesBoosting, clustering, forecasting