Meaning as coordinates
A map gives every city two numbers: latitude and longitude. Cities that are close on the map tend to share weather, culture and travel routes. An embedding does the same for meaning, except it usually uses hundreds or thousands of dimensions instead of two.
“Cat” and “dog” end up closer to each other than either is to “bicycle”. “Refund my order” lands near “I want my money back” even though the two phrases share almost no words. No one assigns those coordinates by hand; a model learns them from patterns of co-occurrence in data.
The numbers are not reversible in any useful sense and they are not a shorter copy of the text. They are coordinates that preserve selected similarities — which similarities depends on what the model was trained to care about.
How embeddings are learned
Early word embeddings such as word2vec were trained with a simple prediction task: given a word, predict its neighbours, or given the neighbours, predict the word. Words that appear in similar contexts end up with similar vectors, which is why the geometry captures a surprising amount of semantic structure.
A transformer produces contextual embeddings: the vector for “bank” in “river bank” differs from the vector in “bank loan”, because each token’s representation is mixed with the tokens around it. Modern embedding models are trained with contrastive objectives — pull a query and its correct passage together, push unrelated passages away — which is precisely the property retrieval needs.
If every vector is normalised to unit length, cosine similarity and the dot product rank results identically. Many vector databases normalise on write for exactly that reason: the dot product is cheaper to compute.
A worked similarity check
Suppose a tiny two-dimensional space gives cat = [0.9, 0.1], dog = [0.8, 0.2] and car = [0.1, 0.9]. The cosine similarity of cat and dog is about 0.99; cat and car is about 0.22. The numbers are arbitrary but the arithmetic is not: the model has placed the two animals almost on top of each other and the vehicle roughly perpendicular.
| Pair | Cosine similarity | What it suggests |
|---|---|---|
| cat ↔ dog | ≈ 0.99 | Nearly the same direction |
| cat ↔ car | ≈ 0.22 | Mostly unrelated |
| dog ↔ car | ≈ 0.35 | Weakly related at best |
The demonstration below plots a handful of example vectors and recomputes these scores as you switch the target word. It is the same calculation a vector database runs at scale.
Where embeddings do real work
Semantic search embeds every document once, embeds the query at request time, and ranks by similarity. Unlike keyword search, it can match “how do I cancel” with “stop my subscription”. Retrieval-augmented generation uses the same mechanism to fetch passages that are placed in the prompt, so the language model answers from supplied text instead of memory.
Recommendation systems embed users and items in one space and recommend the nearest items. Support teams cluster incoming tickets by embedding to find duplicate issues. Vision and audio models use the same idea with different encoders, which is what allows a text query to search an image library.
The engineering details matter more than the geometry in production: chunk size, whether to embed titles with body text, how to combine keyword and vector scores, and when to re-embed after changing models. Vectors from two different models are not comparable, so a model upgrade means re-indexing the corpus.
Common misconceptions and real limitations
- Embeddings do not store facts. They store similarity structure; a vector cannot reliably answer “what is the refund window?” without the original text.
- Closer is not automatically correct. High cosine similarity can reflect shared topic or style rather than the answer you need, which is why reranking and evaluation matter.
- Dimensions are not interpretable features. Individual coordinates rarely map to human concepts, so “the dimension for politeness” is a myth.
- Length carries information in some models. Cosine similarity deliberately ignores magnitude, which can hide confidence or frequency signals.
There is also a cost: embedding a large corpus takes time and money, and storing thousands of floats per chunk adds up. For a small document set, keyword search is often faster, cheaper and easier to explain — embeddings are worth it when paraphrases matter.