Pillar 1 · Model Selection — family deep dive
Types of Embedding Models
Embeddings turn meaning into geometry: similar content lands at nearby points in vector space. The types differ in what they embed (text, images, code) and how they compare (fast one-vector matching vs slow precise reranking). Together they form the retrieval layer under every RAG system and every "find things like this" feature.
The pattern that matters: retrieve fast, rerank precisely
A bi-encoder embeds query and documents separately — comparison is a cheap vector-distance lookup, so it scans millions of documents in milliseconds but only approximately. A cross-encoder (reranker) reads query and document together — far more accurate, far too slow for millions. Production search uses both, in stages:
Two-stage retrieval — the standard search & RAG pipeline
The types, with named models
| Type | Named models | Choose it when | Watch out |
|---|---|---|---|
| Text embeddings (bi-encoder) | OpenAI text-embedding-3-small/-large, Cohere Embed v3, BGE-M3, E5, GTE, Voyage AI | Semantic search, RAG retrieval, dedup, clustering — the default buy | Model change ⇒ re-embed the whole corpus; pin versions |
| Rerankers (cross-encoder) | Cohere Rerank 3, bge-reranker, Jina Reranker, MiniLM cross-encoders | Quality boost on top of any retriever — usually the best accuracy-per-dollar upgrade in RAG | Only rescores candidates — can't rescue a retriever that missed the document |
| Multimodal embeddings | CLIP, SigLIP, OpenCLIP | Text↔image search ("find photos of damaged pallets"), zero-shot image tagging, moderation | Weaker on fine-grained text inside images — that's OCR's job |
| Late-interaction | ColBERT v2 | Middle ground: near cross-encoder quality at near bi-encoder speed | Bigger index; fewer managed offerings |
| Domain-tuned embeddings | voyage-code (code), legal/medical fine-tunes, in-house tuned BGE | Heavy jargon corpora where general embedders confuse terms | Maintenance burden — benchmark against general models first |