Introduction
To a computer, the words "car" and "automobile" are different strings with no letters in common. Yet a model must treat them almost the same. The solution used by every modern language model is the embedding: each token, and later each sentence or document, becomes a list of numbers, learned so that things with similar meanings get similar lists. Embeddings power semantic search, recommendations and the retrieval step of AI assistants.
How it works
An embedding is a vector, such as [0.9, 0.1, 0.0]. Real models learn vectors of a few hundred to a few thousand dimensions during training, and no single dimension has to mean anything a person would name. In this lesson the vectors are made by hand with three readable dimensions, royal, female and animal, so you can follow the numbers.
To compare two vectors a and b:
dot product: a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ
length: |a| = √(a · a)
cosine similarity: cos(a, b) = a · b / (|a| × |b|)
Cosine similarity measures the angle between the vectors: 1 when they point the same way, 0 when they are unrelated (at right angles), −1 when opposite. It ignores length, which is useful because a long document and a short one about the same topic should still match.
The nearest neighbours of a vector are the stored vectors with the highest cosine similarity to it: to find words related to "king", rank every other word by its similarity to king.
Embeddings also capture relations as directions. The step from man to woman is roughly the same as the step from king to queen, so king − man + woman lands near queen. Analogies like this were first noticed in word2vec embeddings (2013); they work only roughly in real models, but they show that the space has structure.
Example
import math
# Hand-made embeddings with three meanings: (royal, female, animal)
emb = {
"king": [0.9, 0.1, 0.0],
"queen": [0.9, 0.6, 0.0],
"man": [0.2, 0.1, 0.1],
"woman": [0.2, 0.9, 0.1],
"cat": [0.0, 0.5, 0.9],
}
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
lengths = round(math.sqrt(sum(x * x for x in a) * sum(y * y for y in b)), 12) # rounded, so every computer agrees
return dot / lengths
for word in ["queen", "man", "cat"]:
print(f"cosine(king, {word}) = {cosine(emb['king'], emb[word]):.3f}")
target = [k - m + w for k, m, w in zip(emb["king"], emb["man"], emb["woman"])]
candidates = [word for word in emb if word not in ("king", "man", "woman")]
best = max(candidates, key=lambda word: cosine(target, emb[word]))
print("king - man + woman =", [round(x, 2) for x in target], "->", best)Line by line
- Each word has a three-number embedding; queen differs from king only in the "female" dimension.
cosinedivides the dot product by the product of the two lengths.- King is closest to queen, a little less close to man, and far from cat, which shares neither royalty nor much else.
targetadds and subtracts the vectors number by number.- Among the words not used in the question, the one most similar to the target is queen.
Expected result
cosine(king, queen) = 0.888
cosine(king, man) = 0.857
cosine(king, cat) = 0.054
king - man + woman = [0.9, 0.9, 0.0] -> queen
In the real world
Search engines, shops and streaming services embed queries, products and songs, then recommend nearest neighbours. AI assistants embed documents so they can find the passages relevant to a question (the next two lessons). Vector databases exist just to store millions of embeddings and find nearest neighbours fast. Embeddings also carry the biases of their training text, for example linking jobs with genders, which is why they are tested before use.
Common mistakes
- Comparing raw dot products of vectors with different lengths, when you mean cosine similarity.
- Mixing embeddings from different models, whose spaces are unrelated.
- Reading meaning into single dimensions of learned embeddings.
- Trusting analogies too much: they hold only approximately.
Best practices
- Normalize vectors to length 1 once; then the dot product is the cosine similarity.
- Use the same embedding model for everything you compare.
- Check nearest neighbours by hand on examples you understand before relying on them.
Summary
An embedding represents a token, word or text as a vector, learned so that similar meanings are close together. Cosine similarity, the dot product divided by the lengths, compares directions. Nearest-neighbour search finds related items, and vector arithmetic can express relations such as king − man + woman ≈ queen.
Quick check
Key takeaways
- Models turn every token into an embedding vector; distances between vectors stand for differences in meaning.
- Cosine similarity compares directions and ignores lengths.
- Vector arithmetic can capture relations, such as king − man + woman ≈ queen.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.