Image generation

Intermediate

Introduction

Text models predict the next token, but how does a machine draw a picture from a sentence? The answer starts with a plain fact: an image is a grid of numbers. A grayscale picture stores one brightness value per pixel, from 0 (black) to 255 (white); a colour picture stores three values, for red, green and blue. Once a picture is numbers, the ideas of the earlier lessons apply: learn the distribution of real images, then sample a new one. Today's best-known image generators use diffusion models, and this lesson shows the idea on images small enough to print in a line.

How it works

Diffusion models turn generation into repeated cleaning. Training has two parts.

  1. The forward process adds noise. Take a real image and, at each step, mix in a little random noise, so the picture gets blurrier and finally becomes pure noise. This process needs no learning: it is a formula, and it produces many pairs of a noisy image and its clean original.
  2. The reverse process removes noise. A neural network is trained to look at a noisy image and estimate the noise, or the clean image, that lies under it. If it is good at small steps, it can be chained.

To generate a picture the program starts from pure noise and repeats: ask the network for its estimate, move the image a little towards it, and continue. After a few dozen steps the noise has become a picture. Randomness enters only through the starting noise, so a different start gives a different image. A text prompt is added as an extra input to the network at every step, which steers the result towards "a red bicycle". To save time, many systems run the steps on a small compressed version of the image (the latent space) and decode the result at the end.

Two older ideas still matter:

  • A GAN has a generator that makes images and a discriminator that tries to tell them from real ones. They improve by competing. GANs are fast but hard to train and can miss whole kinds of images.
  • A VAE has an encoder that squeezes an image into a short latent code and a decoder that rebuilds it. Sampling a random code and decoding it makes a new image. VAEs are stable but give softer images; they are also the compressor used by latent diffusion.

In this lesson the network is replaced by a stand-in: a function that guesses which of three known clean images is closest. A real network learns this from millions of images instead of a list, but the loop around it is the same.

Example

# Tiny 1-D "images" of 6 pixels, each from 0 (dark) to 1 (bright)
training = {"bar": "011110", "edges": "110011", "dot": "001100"}
SHADES = " .:-=+*#%@"
seed = 7


def show(pixels):
    """Draws pixels as characters, darkest " " to brightest "@"."""
    return "[" + "".join(SHADES[min(9, max(0, int(p * 10)))] for p in pixels) + "]"


def noise():
    """A random number in [0, 1) from a linear congruential generator."""
    global seed
    seed = (seed * 1103515245 + 12345) % 2**31
    return seed / 2**31


def denoiser(pixels):
    """A stand-in for a trained network: it guesses the clean image closest to the noisy one."""
    best, best_d = "", 99.0
    for name, clean in training.items():
        d = sum((p - int(q)) * (p - int(q)) for p, q in zip(pixels, clean))
        if d < best_d:
            best, best_d = name, d
    return best


# Forward process (makes the training data): mix in more noise at every step
image = [int(c) for c in training["bar"]]
print("bar      ", show(image))
for t in range(1, 4):
    image = [0.5 * p + 0.5 * noise() for p in image]
    print(f"noise {t}  ", show(image))

# Generation: start from pure noise and remove noise step by step
for sample in (1, 2):
    image = [noise() for _ in range(6)]
    start = show(image)
    for step in range(3):
        guess = denoiser(image)
        image = [p + 0.6 * (int(g) - p) for p, g in zip(image, training[guess])]
    print(f"sample {sample} ", start, "->", show(image), "=", guess)
Open in compiler

Line by line

  1. Each tiny image has six pixels, shown as characters from dark (a space) to bright (@). The three training images are a bar, two edges and a dot.
  2. noise is a seeded random number generator, so the run is the same every time.
  3. The forward process mixes half noise into the bar image three times; the bar slowly disappears.
  4. denoiser is the stand-in network: it finds the training image closest to the noisy input.
  5. Generation starts from six random pixels. At each of three steps the image moves 60 percent of the way towards the guess.
  6. Each sample ends as a clean pattern, the one the random start was closest to. A different seed would give different results.

Expected result

bar       [ @@@@ ]
noise 1   [:**%@ ]
noise 2   [*%=+*=]
noise 3   [##::++]
sample 1  [+: +.@] -> [@@  @@] = edges
sample 2  [+=@+%=] -> [ @@@@ ] = bar

In the real world

Diffusion models power tools that draw from text, edit photos, fill in missing parts and make short videos. Designers use them for concepts, stores for product pictures, and game studios for textures. The same denoising idea is used for audio and for 3D shapes. Because they learn from existing pictures, questions of copyright, consent and fake images are part of using them, which the last lesson discusses.

Common mistakes

  • Thinking the model stores and copies pictures; it learns patterns, though it can repeat training images that appeared very often.
  • Expecting the same picture from the same prompt, when a new random start gives a new result.
  • Forgetting that images are numbers, and that a change of format or size changes the data.
  • Treating the stand-in demo as the real thing: a real denoiser is a large trained network.

Best practices

  • Fix the random seed when you need a repeatable result.
  • Check generated images for errors, such as wrong text or hands, before using them.
  • Label synthetic images as such, and respect the rights of the people and artists in the data.

Summary

An image is a grid of pixel numbers, so models can learn the distribution of images. A diffusion model is trained to remove noise, and generates by starting from pure noise and denoising it step by step, steered by a text prompt. GANs learn by competition between a generator and a discriminator, and VAEs by encoding images into a small latent code and decoding it again.

Quick check

1. How is a grayscale image stored?
2. How does a diffusion model generate a picture?
3. What does the forward process of diffusion do?
4. What do the two networks of a GAN do?

Key takeaways

  • An image is just a grid of numbers, so image models work on arrays of pixels or on compressed versions of them.
  • A diffusion model learns to undo small amounts of noise; generating a picture means denoising pure noise many times.
  • GANs, VAEs and diffusion models are three ways to learn the distribution of images and sample new ones.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.