What generative AI is

Introduction

Chat assistants that write essays, apps that turn a sentence into a picture, tools that compose music or finish your code: these are generative AI. They do not look up answers; they create new text, images or sound that resemble what they were trained on. This course shows how, from the simplest possible generator up to the ideas behind large language models, retrieval, prompting and image generation, all in plain Python you can step through.

How it works

Most of the machine learning you may have met is discriminative: given an example, it predicts a label (spam or not, cat or dog). A generative model goes further. It learns how the training data is distributed, how likely each possible piece of data is, and then samples from that distribution to produce new data.

The simplest generative model of text counts letters. In "banana", "bandana" and "cabana", a is the most common letter, so a well-trained letter model should produce many a's. Generation works by sampling: take a random number between 0 and 1, and walk through the letters adding up their probabilities until the running total passes the random number. Likely letters take up more of the range from 0 to 1, so they are picked more often.

Every generative model, however large, has these two parts:

  1. Training: learn a probability distribution from examples.
  2. Generation: sample from it, again and again.

What changes from model to model is how much structure the distribution captures. Letter counts know nothing about order, so they produce nonsense. A model of which word follows which (lesson 3) produces phrases. A large language model predicts each next piece of text from everything before it, and an image model learns how pixels fit together. The main families:

FamilyGeneratesIdea
Language modelstext, codepredict the next token, again and again
Diffusion modelsimages, audiolearn to remove noise step by step
GANsimagesa generator and a critic train against each other
Variational autoencodersimages, moleculescompress to a small code and decode new codes

Example

words = ["banana", "bandana", "cabana"]

# Training: learn how often each letter appears
counts = {}
for word in words:
    for ch in word:
        counts[ch] = counts.get(ch, 0) + 1
total = sum(counts.values())
letters = sorted(counts)
probs = {ch: counts[ch] / total for ch in letters}
print("learned:", {ch: round(p, 3) for ch, p in probs.items()})


def sample(r):
    """Turn a random number r in [0, 1) into a letter, by walking the running total of probabilities."""
    running = 0.0
    for ch in letters:
        running += probs[ch]
        if r < running:
            return ch
    return letters[-1]


# Generation: one letter per random number
draws = [0.05, 0.62, 0.31, 0.97, 0.48, 0.80]
print("generated:", "".join(sample(r) for r in draws))
Open in compiler

Line by line

  1. Training counts each letter in the three words and divides by the total, giving a probability for each letter.
  2. letters is sorted so the walk through the probabilities always happens in the same order.
  3. sample adds the probabilities one by one; the first letter whose running total passes r is chosen.
  4. Each number in draws stands for one random number. Real programs get these from a random number generator (next lesson); fixed numbers make the result repeatable here.
  5. The generated "word" has the right mix of letters, but no idea of order: the model never learned which letter follows which.

Expected result

learned: {'a': 0.474, 'b': 0.158, 'c': 0.053, 'd': 0.053, 'n': 0.263}
generated: abanbn

In the real world

Large language models such as the ones behind chat assistants are generative models of text trained on huge amounts of writing; image generators are trained on billions of captioned pictures. Businesses use them to draft emails and documents, summarize reports, write code, generate product images and answer customer questions. Every one of these systems still does what this tiny program does: it learned a distribution, and each answer is a sample from it.

Common mistakes

  • Thinking the model looks answers up. It generates them, which is why it can produce fluent text that is false.
  • Expecting the same output every time. Sampling gives different results unless the randomness is fixed.
  • Believing a model knows things outside its training data. It can only reproduce patterns it learned.
  • Confusing fluent with correct. Realistic output is what the training rewards, not truth.

Best practices

  • Ask what distribution a model has learned and from which data before trusting its output.
  • Fix the random seed when you need repeatable results, for tests or demonstrations.
  • Check generated facts against a reliable source.

Summary

Generative AI creates new data by learning the probability distribution of its training data and sampling from it. Discriminative models only separate classes; generative ones can produce examples. Sampling turns a random number into an outcome, more often for likely outcomes. Models differ in how much structure their distribution captures, from letter counts to large language models and image generators.

Quick check

1. What does a generative model learn?
2. Why does a letter-frequency model produce nonsense words?
3. A spam filter that labels emails is...
4. Which family generates images by removing noise step by step?

Key takeaways

  • A generative model learns the distribution of its training data and samples new examples from it.
  • Generation is repeated sampling: the same model can give a different result every time.
  • A model can only produce patterns it learned; the more structure it captures, the more realistic its output.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.