How language models work

Tokens and context windows, probabilities and loss for the next token, and the decoding methods that choose words.

About 4 hours of study

Learning objectives

Lessons

  1. 1

    Tokens and context windows Intermediate

    How language models read text: splitting it into tokens, mapping tokens to numbers, subword pieces for rare words, and the context window that limits how much a model can see at once.

  2. 2

    Next-token prediction Intermediate

    What a language model really computes: a score (logit) for every token, softmax probabilities, the cross-entropy loss used in training, and perplexity as a measure of how surprised a model is.

  3. 3

    Decoding: temperature, top-k, top-p and beam search Intermediate

    How a model chooses each token from its probabilities: greedy decoding, temperature, top-k and nucleus (top-p) sampling, and beam search, with what each one does to creativity and reliability.

Module practice

Module quiz

1. A token is...
2. Softmax turns raw scores into...
3. Which decoding rule always picks the most likely token?
4. Raising the temperature makes the distribution...
5. A model gives every true token probability 0.25. Its perplexity is...
6. With top-k = 2 and probabilities 0.5, 0.3, 0.1 and 0.1, which tokens remain after renormalizing?

‹ Generation as samplingEmbeddings, search and retrieval ›