Attention and transformers
Embeddings and similarity, scaled dot-product and causal self-attention, and the transformer with positional encodings, parameter counts and decoding.
About 5 hours of study
Learning objectives
- Use embeddings and cosine similarity
- Compute attention and causal self-attention
- Describe and size a transformer and decode text
Lessons
- 1
Embeddings Advanced
Represent words and other symbols as dense vectors whose directions carry meaning. Compare embeddings with cosine similarity, find nearest neighbours, solve analogies with vector arithmetic, and see how embedding tables are learned.
- 2
Attention Advanced
Let every position in a sequence look at every other position and take a weighted average of what it finds. Compute scaled dot-product attention, use self-attention and causal masks, and see why attention replaced recurrence.
- 3
Transformers Advanced
Assemble the architecture behind modern language models: token embeddings plus positional encodings, then blocks of multi-head self-attention and feed-forward layers with residual connections and layer normalization, and finally next-token prediction and decoding.
Module practice
- Deep learning: Cosine similarityEasy
- Deep learning: Nearest neighbours of a wordMedium
- Deep learning: Solve analogies with vectorsHard
- Deep learning: Attention scoresEasy
- Deep learning: One attention headMedium
- Deep learning: Causal self-attentionHard
- Deep learning: Sinusoidal positional encodingsEasy
- Deep learning: Count a transformer's parametersMedium
- Deep learning: Greedy decoding and beam searchHard