Generation as sampling

What generative models are, how random numbers pick outcomes, and a first text generator that learns word pairs.

About 3 hours of study

Learning objectives

Lessons

  1. 1

    What generative AI is

    Models that create new text, images and sound: generative versus discriminative models, learning a probability distribution, and sampling new examples from it.

  2. 2

    Sampling and randomness Intermediate

    How a computer makes random choices: pseudo-random number generators and seeds, sampling from a categorical distribution with running totals, and why samples only match their probabilities on average.

  3. 3

    Generating text with a Markov model Intermediate

    The first model that writes phrases: count which word follows which (a bigram model), turn the counts into next-word probabilities, and generate text greedily or by sampling.

Module practice

Module quiz

1. A generative model learns...
2. A linear congruential generator makes random-looking numbers by...
3. In a bigram model, the next word depends on...
4. The probabilities are 0.5, 0.3 and 0.2 and the random number is 0.65. Walking the running total picks...
5. The same seed is used twice with the same sampler. The two runs give...
6. In the training text a word is followed by a 3 times and by b once. What is the probability of a after it?

How language models work ›