Embedding models: the basics
Embedding models turn text into vectors so that meaning becomes something you can measure with distance. A five-minute walkthrough of why we use them, how they're trained, what the major options are (OpenAI, Cohere, Voyage, open-source alternatives), and how to actually pick one.
Embedding models: the basics
A five-minute read.
The previous post treated the embedding model as a black box: text goes in, a list of numbers comes out. That’s fine for understanding vector databases, but the embedding model is doing all of the actual work. The database is just fast storage for its output. Here’s what’s inside the box.
Why we need them at all
A computer can’t compare “puppy” and “young dog” directly. Strings are either equal or not. What we want instead is a representation where closeness in space means closeness in meaning, so that comparison becomes arithmetic: measure a distance, get a similarity.
An embedding model is the thing that produces that representation. It maps any piece of text to a fixed-length vector, and it’s trained so that texts with similar meaning land near each other and unrelated texts land far apart.
flowchart LR
A["good boy"] --> M["embedding model"] --> V1["#91;0.12, -0.41, 0.87, ...#93;"]
B["well-behaved dog"] --> M
M --> V2["#91;0.14, -0.38, 0.83, ...#93;"]
C["quarterly revenue"] --> M --> V3["#91;-0.9, 0.55, -0.02, ...#93;"]V1 and V2 end up close together. V3 ends up far from both. Nobody hand-writes the rules that make that happen; the model learns it.
How one gets trained
Most modern embedding models start from a regular language model — something that already has a decent internal representation of language from pretraining — and then get fine-tuned specifically for the embedding task.
The dominant training signal is contrastive learning: show the model pairs of text that should be close (a question and its answer, two paraphrases, a title and its document) and pairs that should be far apart, then adjust the model until the geometry matches.
flowchart LR
subgraph pairs["training pairs"]
P1["anchor: 'how do I reset my password'"]
P2["positive: 'password reset instructions'"]
P3["negative: 'how do I cancel my order'"]
end
P1 --> M["model"]
P2 --> M
P3 --> M
M --> loss["pull anchor + positive together<br/>push anchor + negative apart"]A pooling step then collapses the model’s internal token-by-token representations into one fixed-length vector — usually by averaging them or by taking a dedicated summary token. That vector is the embedding.
Two architecture details explain most of what differs between models:
- Dimensions. More dimensions can encode more nuance but cost more to store and compare. 384 to 1536 is the common range; some models go higher.
- Context window. How much text it can embed in one pass. Short-context models force you to chunk aggressively; long-context ones let you embed whole pages, though a single vector for a long page still gets blurry, as the previous post mentioned.
What exists right now
Hosted, API-only:
| Model | Notes |
|---|---|
OpenAI text-embedding-3-large / -small |
Widely used default, cheap, good general quality |
Cohere embed-v4 |
Strong multilingual support, built-in compression options |
| Voyage AI | Tuned variants for code and for legal/finance domains |
Google gemini-embedding |
Tightly integrated if you’re already on Google’s stack |
Open source, self-hostable:
| Model | Notes |
|---|---|
| BGE (BAAI) | Consistently near the top of open leaderboards |
| E5 (Microsoft) | Strong general-purpose baseline, several sizes |
| GTE (Alibaba) | Good quality-to-size ratio |
| Nomic Embed | Fully open including training data, long context |
| sentence-transformers (all-MiniLM etc.) | Small, fast, the default “just get something working” pick |
The gap between hosted and open-source has mostly closed for English text. The honest differentiators now are multilingual coverage, context length, licensing, and whether you want the operational cost of running inference yourself versus paying per token.
How to actually choose
Don’t guess from marketing claims. The MTEB leaderboard benchmarks embedding models across retrieval, classification, clustering, and more, and lets you filter by task type, language, and model size. It’s the closest thing this field has to a shared scoreboard.
But treat it as a starting shortlist, not a verdict: MTEB scores are averaged across many tasks and your data won’t look like the benchmark. A model that tops the leaderboard on average can still lose to a smaller one on your specific domain, especially anything jargon-heavy like legal, medical, or code.
A practical order of operations:
- Filter MTEB to your language and a retrieval-shaped task.
- Shortlist two or three candidates spanning a size range.
- Embed a sample of your actual documents and queries with each, and check which one actually retrieves the right passage.
- Only then weigh cost, latency, and self-hosting effort among the finalists.
Two things that bite beginners
Higher dimensions aren’t automatically better. Beyond a point you’re paying more in storage and search latency for retrieval quality that doesn’t move. Some newer models (Matryoshka-trained ones like OpenAI’s v3 and Nomic’s) let you truncate the vector to fewer dimensions with a graceful, predictable quality drop instead of retraining a smaller model.
“State of the art” has a shelf life measured in months. This field moves fast enough that any specific ranking in this post will be stale before long. The MTEB leaderboard, not this list, is the thing to re-check.