Embeddings and similarity

Intermediate

Introduction

To a computer, the words "car" and "automobile" are different strings with no letters in common. Yet a model must treat them almost the same. The solution used by every modern language model is the embedding: each token, and later each sentence or document, becomes a list of numbers, learned so that things with similar meanings get similar lists. Embeddings power semantic search, recommendations and the retrieval step of AI assistants.

How it works

An embedding is a vector, such as [0.9, 0.1, 0.0]. Real models learn vectors of a few hundred to a few thousand dimensions during training, and no single dimension has to mean anything a person would name. In this lesson the vectors are made by hand with three readable dimensions, royal, female and animal, so you can follow the numbers.

To compare two vectors a and b:

dot product:        a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ
length:             |a| = √(a · a)
cosine similarity:  cos(a, b) = a · b / (|a| × |b|)

Cosine similarity measures the angle between the vectors: 1 when they point the same way, 0 when they are unrelated (at right angles), −1 when opposite. It ignores length, which is useful because a long document and a short one about the same topic should still match.

The nearest neighbours of a vector are the stored vectors with the highest cosine similarity to it: to find words related to "king", rank every other word by its similarity to king.

Embeddings also capture relations as directions. The step from man to woman is roughly the same as the step from king to queen, so king − man + woman lands near queen. Analogies like this were first noticed in word2vec embeddings (2013); they work only roughly in real models, but they show that the space has structure.

Example

import math

# Hand-made embeddings with three meanings: (royal, female, animal)
emb = {
    "king": [0.9, 0.1, 0.0],
    "queen": [0.9, 0.6, 0.0],
    "man": [0.2, 0.1, 0.1],
    "woman": [0.2, 0.9, 0.1],
    "cat": [0.0, 0.5, 0.9],
}


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    lengths = round(math.sqrt(sum(x * x for x in a) * sum(y * y for y in b)), 12)  # rounded, so every computer agrees
    return dot / lengths


for word in ["queen", "man", "cat"]:
    print(f"cosine(king, {word}) = {cosine(emb['king'], emb[word]):.3f}")

target = [k - m + w for k, m, w in zip(emb["king"], emb["man"], emb["woman"])]
candidates = [word for word in emb if word not in ("king", "man", "woman")]
best = max(candidates, key=lambda word: cosine(target, emb[word]))
print("king - man + woman =", [round(x, 2) for x in target], "->", best)
Open in compiler

Line by line

  1. Each word has a three-number embedding; queen differs from king only in the "female" dimension.
  2. cosine divides the dot product by the product of the two lengths.
  3. King is closest to queen, a little less close to man, and far from cat, which shares neither royalty nor much else.
  4. target adds and subtracts the vectors number by number.
  5. Among the words not used in the question, the one most similar to the target is queen.

Expected result

cosine(king, queen) = 0.888
cosine(king, man) = 0.857
cosine(king, cat) = 0.054
king - man + woman = [0.9, 0.9, 0.0] -> queen

In the real world

Search engines, shops and streaming services embed queries, products and songs, then recommend nearest neighbours. AI assistants embed documents so they can find the passages relevant to a question (the next two lessons). Vector databases exist just to store millions of embeddings and find nearest neighbours fast. Embeddings also carry the biases of their training text, for example linking jobs with genders, which is why they are tested before use.

Common mistakes

  • Comparing raw dot products of vectors with different lengths, when you mean cosine similarity.
  • Mixing embeddings from different models, whose spaces are unrelated.
  • Reading meaning into single dimensions of learned embeddings.
  • Trusting analogies too much: they hold only approximately.

Best practices

  • Normalize vectors to length 1 once; then the dot product is the cosine similarity.
  • Use the same embedding model for everything you compare.
  • Check nearest neighbours by hand on examples you understand before relying on them.

Summary

An embedding represents a token, word or text as a vector, learned so that similar meanings are close together. Cosine similarity, the dot product divided by the lengths, compares directions. Nearest-neighbour search finds related items, and vector arithmetic can express relations such as king − man + woman ≈ queen.

Quick check

1. What is the cosine similarity of two vectors pointing in exactly the same direction?
2. What is the dot product of [1, 2] and [3, 4]?
3. Why use cosine similarity rather than the raw dot product?
4. What does king − man + woman ≈ queen show?

Key takeaways

  • Models turn every token into an embedding vector; distances between vectors stand for differences in meaning.
  • Cosine similarity compares directions and ignores lengths.
  • Vector arithmetic can capture relations, such as king − man + woman ≈ queen.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.