Attention and transformers

Embeddings and similarity, scaled dot-product and causal self-attention, and the transformer with positional encodings, parameter counts and decoding.

About 5 hours of study

Learning objectives

Lessons

  1. 1

    Embeddings Advanced

    Represent words and other symbols as dense vectors whose directions carry meaning. Compare embeddings with cosine similarity, find nearest neighbours, solve analogies with vector arithmetic, and see how embedding tables are learned.

  2. 2

    Attention Advanced

    Let every position in a sequence look at every other position and take a weighted average of what it finds. Compute scaled dot-product attention, use self-attention and causal masks, and see why attention replaced recurrence.

  3. 3

    Transformers Advanced

    Assemble the architecture behind modern language models: token embeddings plus positional encodings, then blocks of multi-head self-attention and feed-forward layers with residual connections and layer normalization, and finally next-token prediction and decoding.

Module practice

Module quiz

1. What is the cosine similarity of two vectors pointing the same way?
2. What do one query's attention weights add up to?
3. Why are attention scores divided by √d?
4. Why do transformers add positional encodings?
5. What does a causal mask prevent?
6. How can beam search beat greedy decoding?

‹ Convolutional and recurrent networksFinal assessment ›