← All tutorials
Embedding Models

How embedding models work

An embedding model turns a piece of text into a list of numbers β€” a vector β€” positioned so that things with similar meaning land close together. That single trick powers semantic search, RAG, recommendations, clustering and deduplication.

"How do I get a refund?"β†’ embedding modelβ†’ [0.021, -0.44, 0.13, … ]
The one idea. Meaning becomes geometry. Once text is a point in space, "find related text" becomes "find nearby points" β€” something computers do fast, even over millions of items. Everything else in this guide is detail on how that mapping is built and used.

Concept  What is an embedding?

An embedding is a fixed-length vector of floating-point numbers β€” say 384 or 1536 of them β€” that represents a piece of content. The count is the model's dimensionality. Each dimension is a learned axis of meaning; you can't name them, but together the coordinates place the text in a high-dimensional "meaning space."

The defining property: semantic similarity becomes spatial proximity. "How do I get a refund?" and "What is your return policy?" embed to nearby vectors even though they share almost no words, while "The sky is blue" lands far away. Keyword search can't see that the first two mean the same thing; embeddings can.

Embedding vs generation. A chat model produces text; an embedding model produces a vector and generates nothing. They're different tools β€” you often use an embedding model to find the right context, then hand it to a chat model to answer (that's RAG).

Intuition  The semantic space

Real embeddings have hundreds of dimensions, but the idea is visible in 2-D: related concepts cluster, and a search is just "which points are nearest the query?"

A 2-D sketch of embedding space Words with similar meanings cluster together β€” animals, fruits and vehicles form separate groups. A query "a small dog" lands inside the animals cluster, and its nearest neighbours (dog, puppy) are the search results. animals fruits vehicles dog cat puppy query: β€œa small dog” apple banana grape car bus bike

The query "a small dog" never appears in the data, yet it lands next to dog and puppy β€” so a nearest-neighbour lookup returns them. That's semantic search in one picture.

The famous "king βˆ’ man + woman β‰ˆ queen"

Because directions in the space encode relationships, you can sometimes do arithmetic on meaning: the vector from man to woman is roughly parallel to the one from king to queen. Early word2vec embeddings made this vivid; modern sentence embeddings capture far richer, sentence-level meaning.

How  How models learn the space

No one hand-places the points. The model learns where things go from data, and the dominant recipe today is contrastive learning: show the model pairs that should be close and pairs that should be far, and nudge the vectors accordingly.

similar pairpull together adjust vectorspush apart unrelated pair

Training pairs come from naturally-aligned data: a question and its accepted answer, a sentence and its paraphrase, a caption and its image, duplicate FAQ entries. Over millions of such positive pairs (and many negatives), the model tunes its weights until similar meanings reliably land near each other. The loss (e.g. InfoNCE / triplet loss) literally rewards small distances for positives and large ones for negatives.

From word2vec to transformer sentence encoders

Early models (word2vec, GloVe) gave each word one fixed vector β€” so "bank" had a single meaning. Modern embedding models are transformers: they read the whole input, so "river bank" and "savings bank" embed differently. Sentence-Transformers popularized fine-tuning transformers with contrastive objectives to produce strong sentence- and document-level vectors.

Mechanics  From text to a single vector

Inside the model, one call runs this pipeline:

text→ tokenize→ transformer layers→ per-token vectors→ pool→ normalize→ 1 vector
  1. Tokenize β€” split text into subword tokens the model knows.
  2. Transformer β€” attention layers produce a context-aware vector per token (so surrounding words shape each one).
  3. Pool β€” collapse the many token vectors into one, usually by mean-pooling (average them) or taking a special [CLS] token.
  4. Normalize β€” scale the vector to length 1, so only its direction (meaning) matters, not its magnitude.
embed.py β€” text to vectors, locally
# pip install sentence-transformers
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dimensional vectors

texts = ["How do I get a refund?",
         "What is your return policy?",
         "The sky is blue today."]

vecs = model.encode(texts, normalize_embeddings=True)   # pool + L2-normalize
print(vecs.shape)          # (3, 384) β€” one 384-dim vector per text

Compare  Measuring similarity

Two vectors are "similar" when they point the same way. The standard measure is cosine similarity β€” the cosine of the angle between them β€” which runs from -1 (opposite) through 0 (unrelated) to 1 (identical direction).

MetricMeasuresNote
Cosine similarityAngle between vectorsThe default for text; ignores length.
Dot productAngle Γ— magnitudesEquals cosine when vectors are normalized β€” so it's the fast path.
Euclidean (L2)Straight-line distanceSmaller = closer; equivalent to cosine on normalized vectors.
similarity.py
from sentence_transformers import util

# reuse `vecs` from the previous snippet (already normalized)
print(util.cos_sim(vecs[0], vecs[1]))   # β‰ˆ 0.63  refund ↔ returns (related)
print(util.cos_sim(vecs[0], vecs[2]))   # β‰ˆ 0.04  refund ↔ blue sky (unrelated)
Why normalize? After L2-normalization every vector has length 1, so dot product == cosine similarity. Vector databases exploit this: normalize once, then use fast dot-product search. Just be consistent β€” normalize both what you store and what you query.

Practice  Choosing & using an embedding model

Models differ in quality, size, speed, cost and language/domain coverage. A few rules keep you out of trouble.

Picking a model

Do Start from a benchmark like MTEB (the Massive Text Embedding Benchmark) and pick for your task β€” retrieval, similarity, or clustering are scored separately. Weigh dimensions (quality vs storage/compute), max input length (can it hold your chunk?), language/domain coverage, and local vs hosted.

ModelDimsWhereGood for
all-MiniLM-L6-v2384Local (OSS)Fast, cheap, strong baseline
BGE / E5 / GTE384–1024Local (OSS)Top open retrieval; use query/passage prefixes
OpenAI text-embedding-3-small1536*Hosted APIStrong, inexpensive, no infra
text-embedding-3-large3072*Hosted APIHighest quality when it matters
Voyage / CoherevariesHosted APIRetrieval- & domain-tuned options

* Some models support Matryoshka truncation β€” shorten the vector (e.g. 1536 β†’ 256) to trade a little accuracy for big storage/speed savings.

Using it well β€” the rules that bite

Trap Mixing models. Vectors from different models live in different spaces and are not comparable β€” you must embed your index and your queries with the same model. Change the model β†’ re-embed everything.

Trap Asymmetric tasks. Search matches a short query against long documents; use a retrieval-tuned model (often with "query:" / "passage:" prefixes), not a plain similarity model.

Trap Chunks too big. One vector must summarize the whole input; embed an entire document and the meaning blurs. Chunk to focused passages (see RAG). Text past the model's max length is silently truncated.

Do Normalize, cache, and expect fuzziness. Normalize vectors; cache them (re-embedding is the cost); and remember embeddings capture meaning, not exact tokens β€” pair with keyword/hybrid search when exact IDs or rare terms matter.

Wider view  Beyond text

The same idea generalizes to any content you can train alignment on:

πŸ–ΌοΈ Multimodal

Models like CLIP embed images and text into one shared space β€” so you can search photos with a text query.

πŸ’» Code

Code-specific embeddings map functions and docstrings together, powering semantic code search.

🌐 Multilingual

Multilingual models place the same meaning from different languages nearby, enabling cross-language search.

Same trick everywhere. Whether it's text, images, code or audio, an embedding model's job is identical: map inputs to vectors where distance means dissimilarity. Master it once and it transfers.

Cheat sheet

QuestionShort answer
What does it output?A fixed-length vector (e.g. 384–3072 floats), not text.
What makes two texts "similar"?A small angle between their vectors β†’ high cosine similarity.
How is the space learned?Contrastive training: pull similar pairs together, push others apart.
Which similarity metric?Cosine β€” and dot product once vectors are normalized.
Biggest footgun?Using different models for indexing vs querying.
When embeddings aren't enough?Exact keywords/IDs β€” add hybrid (keyword + vector) search.

Knowledge check

Six questions on how embeddings work. Pick an answer for each, then check your score.

1. What does an embedding model output for a piece of text?

2. Two texts are semantically similar when their vectors…

3. How do modern embedding models learn where to place things?

4. Why do we usually normalize embeddings to length 1?

5. What's the classic mistake when building a search index?

6. Your users search by exact product IDs and rare codes. Embeddings alone struggle β€” what helps?

Next steps

πŸ”Ž RAG vs CAG vs GraphRAG

Retrieval in depth β€” embeddings are the engine behind RAG; see when to retrieve vs cache.

πŸ—„οΈ Vector databases

Where vectors live, compared β€” Chroma, FAISS, Qdrant, Pinecone, pgvector & more, and how they search.

πŸ€– LLM apps tutorial

Build a RAG chain that embeds, stores and retrieves your documents.

πŸ““ All tutorials

The hub β€” LLM apps, retrieval, agent kits, AWS services, and the IoT example.