Introduction
Networks compute with numbers, but much of the world comes as symbols: words, product IDs, user IDs, amino acids. The simplest encoding, a one-hot vector, makes every symbol equally different from every other: "cat" is as far from "kitten" as from "carburettor". Embeddings fix this by giving each symbol a short dense vector, learned so that related symbols get similar vectors. Every language model starts by looking up an embedding for each token.
How it works
An embedding table E has one row per item in the vocabulary and d columns, often a few hundred. Looking up an item returns its row. That is the same as multiplying its one-hot vector by E, but much faster, so a lookup is really a linear layer without the arithmetic.
The vectors start random and are trained by backpropagation with the rest of the network. Training pushes items used in similar contexts towards similar vectors, following the distributional hypothesis: "you shall know a word by the company it keeps". Word2vec, for example, learned word vectors by predicting a word's neighbours in large amounts of text.
Cosine similarity compares two embeddings by the angle between them:
cos(a, b) = (a · b) / (|a| |b|)
It is 1 for vectors pointing the same way, 0 for perpendicular ones and −1 for opposite ones, and it ignores length. The nearest neighbours of a word, the words with the highest cosine similarity, are usually related words.
Directions in a good embedding space can carry meaning. The classic example is analogy by vector arithmetic: king − man + woman lands near queen, because the difference between "man" and "woman" is roughly the same direction as between "king" and "queen". To answer "a is to b as c is to ?", compute b − a + c and find the nearest word, excluding a, b and c themselves.
Syntax
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
target = [vb - va + vc for va, vb, vc in zip(E[a], E[b], E[c])]
Example
import math
vectors = {
"king": [0.9, 0.9, 0.1, 0.0],
"queen": [0.9, 0.1, 0.9, 0.0],
"man": [0.1, 0.9, 0.1, 0.0],
"woman": [0.1, 0.1, 0.9, 0.0],
"prince": [0.7, 0.8, 0.1, 0.0],
"apple": [0.0, 0.1, 0.1, 0.9],
"banana": [0.0, 0.0, 0.1, 0.9],
}
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
for word in ["king", "apple"]:
ranked = sorted((w for w in vectors if w != word), key=lambda w: -cosine(vectors[word], vectors[w]))
print(f"closest to {word}: " + ", ".join(f"{w} {cosine(vectors[word], vectors[w]):.3f}" for w in ranked[:3]))
target = [k - m + w for k, m, w in zip(vectors["king"], vectors["man"], vectors["woman"])]
best = max((w for w in vectors if w not in ("king", "man", "woman")), key=lambda w: cosine(target, vectors[w]))
print("king - man + woman is closest to:", best)
vocabulary = list(vectors)
one_hot = [1 if w == "queen" else 0 for w in vocabulary]
lookup = [sum(one_hot[i] * vectors[vocabulary[i]][j] for i in range(len(vocabulary))) for j in range(4)]
print("one-hot vector times the embedding table gives the row for queen:", lookup)Line by line
- Each word has a four-number vector; in this toy table the numbers roughly mean royal, male, female and fruit.
- The closest words to "king" are other royal or male words, and the closest to "apple" is "banana".
- king − man + woman removes "male", adds "female" and lands exactly on queen's vector.
- Multiplying a one-hot vector by the table picks out one row, which is what an embedding lookup does.
- In real trained embeddings the dimensions are not this easy to read, but similar items still end up close together.
Expected result
closest to king: prince 0.998, man 0.782, queen 0.607
closest to apple: banana 0.994, man 0.120, woman 0.120
king - man + woman is closest to: queen
one-hot vector times the embedding table gives the row for queen: [0.9, 0.1, 0.9, 0.0]
In the real world
Language models embed tokens before anything else; recommendation systems embed users and products and recommend items whose vectors are close to the user's; search engines embed documents and queries to find matches by meaning rather than exact words (vector search). Embeddings can also reflect biases in their training text, which is an active area of research and an important consideration in applications.
Common mistakes
- Comparing embeddings by raw dot product when lengths differ a lot; normalize or use cosine.
- Forgetting to exclude the query words when solving an analogy; they are often the closest.
- Expecting each dimension to have a clear meaning; in learned embeddings meaning is spread across many dimensions.
- Using one-hot vectors for large vocabularies, which are huge and carry no similarity.
Best practices
- Normalize embeddings to length 1 when you will mostly compare them by cosine.
- Inspect nearest neighbours of a few items to check that training produced sensible embeddings.
- Choose the embedding size by vocabulary size and data: tens for small problems, hundreds or thousands for language.
Summary
Embeddings give each symbol a dense vector stored in a table and learned by backpropagation. Cosine similarity measures how alike two embeddings are, nearest neighbours reveal related items, and vector arithmetic such as king − man + woman can solve analogies.
Quick check
Key takeaways
- Embeddings turn symbols into vectors that networks can compute with, placing similar items close together.
- Cosine similarity compares directions and ignores length.
- Embeddings are ordinary weights, learned by backpropagation like any others.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.