Introduction
To a bag-of-words model, "cat" and "kitten" are as different as "cat" and "carburettor": each word is just a separate column. Word embeddings fix this by giving every word a short list of numbers, perhaps 100 or 300 of them, arranged so that words with similar meanings get similar vectors. Embeddings were a turning point in NLP around 2013, and they are the first layer of every modern neural language model, including large language models.
How it works
A one-hot vector has one position per vocabulary word: "cat" is 1 at its own position and 0 elsewhere. Every pair of different words is equally far apart, so one-hot vectors carry no notion of meaning. An embedding is a dense vector in which every number is used, and directions encode meaning.
Where do the numbers come from? The distributional hypothesis says that a word is characterized by the company it keeps: "tea" and "coffee" both appear near "drink", "cup" and "hot". So:
- Count-based vectors: build a co-occurrence matrix that counts, for each word, how often every other word appears within a small window around it. Each row is a vector for that word. Methods such as PMI weighting and SVD compress these rows into short dense vectors (GloVe is a well-known refinement).
- Prediction-based vectors: word2vec trains a tiny neural network to predict a word from its neighbours (or the neighbours from the word); the learned weights are the embeddings.
To compare vectors, use cosine similarity: cos(u, v) = (u · v) ÷ (‖u‖ × ‖v‖). It is 1 for the same direction, 0 for unrelated directions and −1 for opposite ones, and it ignores length, which mostly reflects how frequent a word is. The nearest neighbours of a word are the words with the highest cosine similarity to it.
Embeddings famously support analogies: to answer "man is to king as woman is to ?", compute king − man + woman and find the nearest word, excluding the three question words. Subtracting "man" removes the male direction, adding "woman" adds the female one, and what is left points towards "queen". This works for some relations (capitals, plurals, verb tenses) and fails for many others.
Example
import math
vectors = {
"king": [0.9, 0.8, 0.0],
"queen": [0.9, -0.8, 0.0],
"man": [0.1, 0.8, 0.0],
"woman": [0.1, -0.8, 0.0],
"prince": [0.8, 0.5, 0.1],
"apple": [0.0, 0.0, 0.9],
"banana": [0.1, 0.0, 0.8],
}
def cosine(u, v):
dot = sum(a * b for a, b in zip(u, v))
return round(dot / (math.sqrt(sum(a * a for a in u)) * math.sqrt(sum(b * b for b in v))), 12)
others = [w for w in vectors if w != "king"]
ranked = sorted(others, key=lambda w: -cosine(vectors["king"], vectors[w]))
print("nearest to king:", [(w, round(cosine(vectors["king"], vectors[w]), 3)) for w in ranked[:3]])
print("apple vs banana:", round(cosine(vectors["apple"], vectors["banana"]), 3))
target = [k - m + w for k, m, w in zip(vectors["king"], vectors["man"], vectors["woman"])]
print("king - man + woman =", [round(x, 2) for x in target])
candidates = [w for w in vectors if w not in ("king", "man", "woman")]
answer = max(candidates, key=lambda w: cosine(target, vectors[w]))
print("closest word:", answer, round(cosine(target, vectors[answer]), 3))Line by line
- Each word has a hand-made 3-number vector. Think of the dimensions as "royal", "male minus female" and "fruit"; real embeddings learn their dimensions, which are rarely so readable.
cosinedivides the dot product by both lengths, so only directions matter. The result is rounded so it is identical on every computer.- The nearest words to "king" are "prince" and "man", which point in a similar direction; "queen" scores low because of the gender dimension.
- "apple" and "banana" are almost parallel (0.992): both are mostly "fruit".
- king − man + woman gives [0.9, −0.8, 0.0], exactly the vector of "queen", so the closest remaining word is "queen" with cosine 1.0.
Expected result
nearest to king: [('prince', 0.98), ('man', 0.752), ('queen', 0.117)]
apple vs banana: 0.992
king - man + woman = [0.9, -0.8, 0.0]
closest word: queen 1.0
In the real world
Pretrained embeddings such as word2vec, GloVe and fastText were trained on billions of words and can be downloaded. With gensim (shown, not run: it is not available in StepVioz's browser Python), training your own looks like this:
from gensim.models import Word2Vec
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)
print(model.wv.most_similar("king", topn=3))
print(model.wv.most_similar(positive=["king", "woman"], negative=["man"], topn=1))
where sentences is a list of token lists. Modern models go further with contextual embeddings: a transformer gives "bank" a different vector in "river bank" and "bank account". Embeddings of whole sentences power semantic search and retrieval for chatbots, still compared by cosine similarity. Embeddings also learn bias from their text: early word2vec vectors put "doctor" closer to "man" and "nurse" closer to "woman", which matters when they are used to rank people.
Common mistakes
- Comparing raw dot products, which favour frequent words with long vectors; use cosine.
- Forgetting to exclude the question words in analogies, which often return one of them.
- Expecting analogies to work for every relation; many published examples were chosen because they worked.
- Using embeddings trained on one domain (news) for another (medicine) without checking.
Best practices
- Normalize vectors to length 1 once; then cosine similarity is just a dot product.
- Inspect nearest neighbours of a few words to check that the embeddings make sense.
- Prefer contextual embeddings from a pretrained transformer when word senses matter.
- Test embeddings for social bias before using them in decisions about people.
Summary
Word embeddings are dense vectors in which similar words point in similar directions. They come from co-occurrence counts or from training a model to predict neighbours, following the distributional hypothesis. Cosine similarity compares them and finds nearest neighbours, and vector arithmetic such as king − man + woman can solve some analogies. Modern models use contextual embeddings, and all embeddings inherit the biases of their training text.
Quick check
Key takeaways
- Embeddings place words with similar meanings close together, which counts and one-hot vectors cannot do.
- Context defines meaning: vectors built or trained from neighbouring words capture similarity.
- Vector arithmetic captures some relations, but embeddings also absorb the biases of their training text.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.