Introduction
A recurrent network passes information along a sequence one step at a time, so a word at the start of a long paragraph has to survive dozens of updates to influence a word at the end. Attention lets every position look directly at every other position and decide how much each one matters. Introduced for translation and then made the centre of the transformer, attention is the core operation of today's language, vision and speech models.
How it works
Each position produces three vectors from its input, using learned matrices:
- a query q: what this position is looking for;
- a key k: what this position offers to others;
- a value v: the information it passes on.
To compute the output for one query:
- scores: take the dot product of the query with every key and divide by √d, where d is the key length: sⱼ = q · kⱼ / √d;
- weights: apply softmax to the scores, so they are positive and sum to 1;
- output: the weighted average of the values, Σⱼ wⱼ vⱼ.
A key similar to the query gets a high score and a large weight. Dividing by √d matters: dot products of random d-dimensional vectors grow like √d, and without the scaling the softmax would put almost all the weight on one position, giving vanishing gradients.
In self-attention the queries, keys and values all come from the same sequence, so each word gathers information from the other words of its own sentence: "it" can attend to the noun it refers to. For predicting the next word, a position must not see words after it, so a causal mask restricts each position to itself and earlier positions.
Attention computes all positions in parallel, which suits graphics processors, and connects any two positions in one step. Multi-head attention runs several attention operations side by side with different learned projections, so different heads can track different relationships. Its cost grows with the square of the sequence length, which is why long contexts are expensive.
Syntax
scores = [sum(a * b for a, b in zip(q, k)) / math.sqrt(len(k)) for k in keys]
weights = softmax(scores)
output = [sum(w * v[c] for w, v in zip(weights, values)) for c in range(len(values[0]))]
Example
import math
def softmax(z):
top = max(z)
e = [round(math.exp(v - top), 12) for v in z] # rounded, so every computer gives the same digits
return [v / sum(e) for v in e]
def attention(queries, keys, values, causal=False):
d = len(keys[0])
outputs, all_weights = [], []
for i, q in enumerate(queries):
visible = range(i + 1) if causal else range(len(keys))
scores = [sum(a * b for a, b in zip(q, keys[j])) / math.sqrt(d) for j in visible]
weights = softmax(scores)
outputs.append([sum(w * values[j][c] for w, j in zip(weights, visible)) for c in range(len(values[0]))])
all_weights.append(weights)
return outputs, all_weights
tokens = ["the", "cat", "sat"]
X = [[1.0, 0.0, 1.0], [0.0, 2.0, 0.0], [1.0, 1.0, 1.0]]
out, weights = attention(X, X, X)
print("full self-attention weights:")
for t, w in zip(tokens, weights):
print(f" {t:4}", " ".join(f"{v:.3f}" for v in w))
out, weights = attention(X, X, X, causal=True)
print("causal self-attention weights (no looking ahead):")
for t, w in zip(tokens, weights):
print(f" {t:4}", " ".join(f"{v:.3f}" for v in w))
print("output for 'sat':", [round(v, 3) for v in out[2]])Line by line
- Three tokens have small vectors; here the same vectors serve as queries, keys and values, which keeps the example short.
- In full self-attention, "cat", whose vector is longest, gets a large weight from itself.
- Each row of weights sums to 1; it says how much each token contributes to that token's output.
- With a causal mask, "the" can only attend to itself, "cat" to "the" and "cat", and "sat" to all three.
- The output for "sat" is the weighted average of the three value vectors.
Expected result
full self-attention weights:
the 0.432 0.136 0.432
cat 0.070 0.707 0.223
sat 0.264 0.264 0.471
causal self-attention weights (no looking ahead):
the 1.000
cat 0.090 0.910
sat 0.264 0.264 0.471
output for 'sat': [0.736, 1.0, 0.736]
In the real world
Every transformer layer in models such as GPT, BERT and their successors is built around multi-head self-attention; vision transformers apply it to image patches. Machine translation models use cross-attention, where the queries come from the sentence being written and the keys and values from the sentence being translated. Engineering work such as FlashAttention speeds up the same mathematics by organizing the memory accesses carefully.
Common mistakes
- Forgetting the √d scaling, which makes the softmax far too sharp.
- Applying softmax over the wrong axis: weights must sum to 1 over the keys for each query.
- Leaving out the causal mask when training a model to predict the next token, so it can cheat by looking ahead.
- Mixing up keys and values: keys decide the weights, values are what gets averaged.
Best practices
- Check that every row of attention weights sums to 1.
- Print attention weights for a short example; they show what the model looks at.
- Use the causal mask for generation and full attention for understanding tasks.
Summary
Attention compares a query with every key by scaled dot product, turns the scores into weights with softmax, and averages the values with those weights. Self-attention lets every position of a sequence gather information from every other, a causal mask hides the future, and multi-head attention runs several of these in parallel.
Quick check
Key takeaways
- Attention output = Σ softmax(q · k / √d) × v: a weighted average chosen by similarity.
- Any position can attend directly to any other, however far apart they are.
- Dividing by √d keeps scores from growing with the dimension and saturating the softmax.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.