How embedding models work
An embedding model turns a piece of text into a list of numbers β a vector β positioned so that things with similar meaning land close together. That single trick powers semantic search, RAG, recommendations, clustering and deduplication.
Concept What is an embedding?
An embedding is a fixed-length vector of floating-point numbers β say 384 or 1536 of them β that represents a piece of content. The count is the model's dimensionality. Each dimension is a learned axis of meaning; you can't name them, but together the coordinates place the text in a high-dimensional "meaning space."
The defining property: semantic similarity becomes spatial proximity. "How do I get a refund?" and "What is your return policy?" embed to nearby vectors even though they share almost no words, while "The sky is blue" lands far away. Keyword search can't see that the first two mean the same thing; embeddings can.
Intuition The semantic space
Real embeddings have hundreds of dimensions, but the idea is visible in 2-D: related concepts cluster, and a search is just "which points are nearest the query?"
The query "a small dog" never appears in the data, yet it lands next to dog and puppy β so a nearest-neighbour lookup returns them. That's semantic search in one picture.
The famous "king β man + woman β queen"
Because directions in the space encode relationships, you can sometimes do arithmetic on meaning: the vector from man to woman is roughly parallel to the one from king to queen. Early word2vec embeddings made this vivid; modern sentence embeddings capture far richer, sentence-level meaning.
How How models learn the space
No one hand-places the points. The model learns where things go from data, and the dominant recipe today is contrastive learning: show the model pairs that should be close and pairs that should be far, and nudge the vectors accordingly.
Training pairs come from naturally-aligned data: a question and its accepted answer, a sentence and its paraphrase, a caption and its image, duplicate FAQ entries. Over millions of such positive pairs (and many negatives), the model tunes its weights until similar meanings reliably land near each other. The loss (e.g. InfoNCE / triplet loss) literally rewards small distances for positives and large ones for negatives.
From word2vec to transformer sentence encoders
Early models (word2vec, GloVe) gave each word one fixed vector β so "bank" had a single meaning. Modern embedding models are transformers: they read the whole input, so "river bank" and "savings bank" embed differently. Sentence-Transformers popularized fine-tuning transformers with contrastive objectives to produce strong sentence- and document-level vectors.
Mechanics From text to a single vector
Inside the model, one call runs this pipeline:
- Tokenize β split text into subword tokens the model knows.
- Transformer β attention layers produce a context-aware vector per token (so surrounding words shape each one).
- Pool β collapse the many token vectors into one, usually by mean-pooling (average them) or taking a special
[CLS]token. - Normalize β scale the vector to length 1, so only its direction (meaning) matters, not its magnitude.
# pip install sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dimensional vectors
texts = ["How do I get a refund?",
"What is your return policy?",
"The sky is blue today."]
vecs = model.encode(texts, normalize_embeddings=True) # pool + L2-normalize
print(vecs.shape) # (3, 384) β one 384-dim vector per text
Compare Measuring similarity
Two vectors are "similar" when they point the same way. The standard measure is cosine similarity β the cosine of the angle between them β which runs from -1 (opposite) through 0 (unrelated) to 1 (identical direction).
| Metric | Measures | Note |
|---|---|---|
| Cosine similarity | Angle between vectors | The default for text; ignores length. |
| Dot product | Angle Γ magnitudes | Equals cosine when vectors are normalized β so it's the fast path. |
| Euclidean (L2) | Straight-line distance | Smaller = closer; equivalent to cosine on normalized vectors. |
from sentence_transformers import util
# reuse `vecs` from the previous snippet (already normalized)
print(util.cos_sim(vecs[0], vecs[1])) # β 0.63 refund β returns (related)
print(util.cos_sim(vecs[0], vecs[2])) # β 0.04 refund β blue sky (unrelated)
Practice Choosing & using an embedding model
Models differ in quality, size, speed, cost and language/domain coverage. A few rules keep you out of trouble.
Picking a model
Do Start from a benchmark like MTEB (the Massive Text Embedding Benchmark) and pick for your task β retrieval, similarity, or clustering are scored separately. Weigh dimensions (quality vs storage/compute), max input length (can it hold your chunk?), language/domain coverage, and local vs hosted.
| Model | Dims | Where | Good for |
|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | Local (OSS) | Fast, cheap, strong baseline |
| BGE / E5 / GTE | 384β1024 | Local (OSS) | Top open retrieval; use query/passage prefixes |
| OpenAI text-embedding-3-small | 1536* | Hosted API | Strong, inexpensive, no infra |
| text-embedding-3-large | 3072* | Hosted API | Highest quality when it matters |
| Voyage / Cohere | varies | Hosted API | Retrieval- & domain-tuned options |
* Some models support Matryoshka truncation β shorten the vector (e.g. 1536 β 256) to trade a little accuracy for big storage/speed savings.
Using it well β the rules that bite
Trap Mixing models. Vectors from different models live in different spaces and are not comparable β you must embed your index and your queries with the same model. Change the model β re-embed everything.
Trap Asymmetric tasks. Search matches a short query against long documents; use a retrieval-tuned model (often with "query:" / "passage:" prefixes), not a plain similarity model.
Trap Chunks too big. One vector must summarize the whole input; embed an entire document and the meaning blurs. Chunk to focused passages (see RAG). Text past the model's max length is silently truncated.
Do Normalize, cache, and expect fuzziness. Normalize vectors; cache them (re-embedding is the cost); and remember embeddings capture meaning, not exact tokens β pair with keyword/hybrid search when exact IDs or rare terms matter.
Wider view Beyond text
The same idea generalizes to any content you can train alignment on:
πΌοΈ Multimodal
Models like CLIP embed images and text into one shared space β so you can search photos with a text query.
π» Code
Code-specific embeddings map functions and docstrings together, powering semantic code search.
π Multilingual
Multilingual models place the same meaning from different languages nearby, enabling cross-language search.
Cheat sheet
| Question | Short answer |
|---|---|
| What does it output? | A fixed-length vector (e.g. 384β3072 floats), not text. |
| What makes two texts "similar"? | A small angle between their vectors β high cosine similarity. |
| How is the space learned? | Contrastive training: pull similar pairs together, push others apart. |
| Which similarity metric? | Cosine β and dot product once vectors are normalized. |
| Biggest footgun? | Using different models for indexing vs querying. |
| When embeddings aren't enough? | Exact keywords/IDs β add hybrid (keyword + vector) search. |
Knowledge check
Six questions on how embeddings work. Pick an answer for each, then check your score.
Next steps
π RAG vs CAG vs GraphRAG
Retrieval in depth β embeddings are the engine behind RAG; see when to retrieve vs cache.
ποΈ Vector databases
Where vectors live, compared β Chroma, FAISS, Qdrant, Pinecone, pgvector & more, and how they search.
π€ LLM apps tutorial
Build a RAG chain that embeds, stores and retrieves your documents.
π All tutorials
The hub β LLM apps, retrieval, agent kits, AWS services, and the IoT example.