Generators

Introduction

A generator is a function that gives back values one at a time. Instead of return, it uses yield, which hands out a value and pauses the function until the next value is wanted.

Making a list means holding every item in memory at once, which is impossible for an endless sequence and wasteful for a huge one. A generator produces one value at a time, only when it is asked.

How it works

Endless sequences

def squares():
    n = 1
    while True:
        yield n * n
        n += 1

gen = squares()
print(next(gen), next(gen), next(gen))

squares() has while True, yet the program finishes: a generator only runs when asked. next(gen) asks for one value at a time. This is how programs handle huge or endless data, such as reading a big file line by line, without holding it all in memory.

Generator expressions

total = sum(n * n for n in range(1, 11))
print(total)

Round brackets around a comprehension make a generator instead of a list. sum(...) pulls the values one by one, so no list of squares is ever built.

Syntax

def count_up(limit):
    n = 1
    while n <= limit:
        yield n
        n += 1

for value in count_up(3):
    print(value)

total = sum(n * n for n in range(10))

Example

def count_up(limit):
    n = 1
    while n <= limit:
        yield n
        n += 1

for value in count_up(3):
    print(value)

Step through slowly and watch the call stack. Each time the loop asks for a value, a count_up frame appears, runs until yield, and disappears again, but its local n is remembered for next time. The frame comes back for 1, 2 and 3; when n passes the limit, the generator ends and so does the loop.

Line by line

1. def count_up(limit): contains yield, so calling it creates a generator and runs nothing yet.
2. The for loop asks for the first value, and the body runs until yield n hands out 1.
3. print(value) shows it, and the generator stays paused with n remembered.
4. The next round carries on after the yield, adding 1 to n.
5. When n passes the limit, the function ends, and so does the loop.

Expected result

1
2
3

In the real world

Reading a very large file one line at a time, streaming data from a sensor and handling a long list of results from a database all rely on this idea. Python's range and zip are lazy for the same reason.

Common mistakes

• Calling a generator function and expecting a result: count_up(3) returns a generator object; loop over it or call next.
• Looping over the same generator twice: the second loop gets nothing, because it is used up.
• Calling next on a finished generator raises StopIteration.

Best practices

• Use a generator when you only need to go through the values once.
• Convert to a list only when you need to index or reuse the values.
• Keep generators simple, with one clear job.

Summary

A generator uses yield to hand out values one at a time, pausing between them. Generators allow endless sequences and use little memory, but each can only be used once.

Practice

Write a generator evens(limit) that yields 2, 4, 6, … up to limit, and use it in a for loop.

Mark lesson complete