How language models work
Tokens and context windows, probabilities and loss for the next token, and the decoding methods that choose words.
About 4 hours of study
Learning objectives
- Split text into tokens and fit a context window
- Compute softmax, loss and perplexity
- Choose tokens with temperature, top-k, top-p and beam search
Lessons
- 1
Tokens and context windows Intermediate
How language models read text: splitting it into tokens, mapping tokens to numbers, subword pieces for rare words, and the context window that limits how much a model can see at once.
- 2
Next-token prediction Intermediate
What a language model really computes: a score (logit) for every token, softmax probabilities, the cross-entropy loss used in training, and perplexity as a measure of how surprised a model is.
- 3
Decoding: temperature, top-k, top-p and beam search Intermediate
How a model chooses each token from its probabilities: greedy decoding, temperature, top-k and nucleus (top-p) sampling, and beam search, with what each one does to creativity and reliability.
Module practice
- Words to token IDsEasy
- Subword tokens by longest matchMedium
- Fit a chat into the context windowHard
- Softmax over logitsEasy
- Probability and loss of the true tokenMedium
- Perplexity of a sentenceHard
- Apply a temperatureEasy
- Top-k and top-p filteringMedium
- Beam search over a bigram modelHard