Generation as sampling
What generative models are, how random numbers pick outcomes, and a first text generator that learns word pairs.
About 3 hours of study
Learning objectives
- Separate generative from discriminative models
- Sample from weighted outcomes with seeded random numbers
- Generate text with a bigram model
Lessons
- 1
What generative AI is
Models that create new text, images and sound: generative versus discriminative models, learning a probability distribution, and sampling new examples from it.
- 2
Sampling and randomness Intermediate
How a computer makes random choices: pseudo-random number generators and seeds, sampling from a categorical distribution with running totals, and why samples only match their probabilities on average.
- 3
Generating text with a Markov model Intermediate
The first model that writes phrases: count which word follows which (a bigram model), turn the counts into next-word probabilities, and generate text greedily or by sampling.
Module practice
- Learn letter probabilitiesEasy
- Generate letters from a learned modelMedium
- Which text did it come from?Hard
- A linear congruential generatorEasy
- Sample from weighted outcomesMedium
- A seeded sampler with expected countsHard
- Count word pairsEasy
- Next-word probabilitiesMedium
- Generate text with a bigram modelHard