🎓 A1Learner
🔍

Python Generator Patterns

A generator pipeline is a chain where each stage feeds the next stage.

The source stage

The source makes values from nothing. It just yields them one by one.

def read_numbers():
    yield 2
    yield 3
    yield 4

print(list(read_numbers()))

The transform stage

A transform takes a stream in. It yields each value after a change.

def doubled(stream):
    for value in stream:
        yield value * 2

print(list(doubled([2, 3, 4])))

Chain the stages

You chain stages by nesting calls. Read the chain from the inside out.

def read_numbers():
    yield 2
    yield 3
    yield 4

def doubled(stream):
    for value in stream:
        yield value * 2

def add_one(stream):
    for value in stream:
        yield value + 1

print(list(add_one(doubled(read_numbers()))))

Flatten nested data

yield from opens each inner group. Its values come out one by one.

def flatten(groups):
    for group in groups:
        yield from group

data = [[1, 2], [3], [4, 5]]
print(list(flatten(data)))

The sink pulls

The sink is plain code that consumes. This for loop is the sink.

def shout(stream):
    for line in stream:
        yield line.upper()

for line in shout(["hi", "bye"]):
    print(line)

Example

def read_lines(lines):
    for line in lines:
        yield line

def clean(stream):
    for line in stream:
        yield line.strip().lower()

def keep_errors(stream):
    for line in stream:
        if line.startswith("error"):
            yield line

raw_lines = [
    "  ERROR: disk full  ",
    "info: started",
    " ERROR: time out ",
    "info: done",
    "warning: slow disk",
]

for line in keep_errors(clean(read_lines(raw_lines))):
    print(line)

Key points

  • Pipeline: a chain where each stage feeds the next stage
  • Source: the first stage, which makes the starting values
  • Transform: a middle stage, which changes each value
  • Sink: the last stage, which pulls values and uses them
  • Use yield from to flatten one level of nested data

In this example, read_lines yields each raw line. clean takes that stream in. It strips spaces from each line. It also makes each line lower case. keep_errors takes the clean stream in. It yields only lines that start with error. The for loop is the sink. It pulls the chain and prints each line that survives.

Note

In Python, a pipeline stage does no work until the sink pulls.

Nothing is built as a big list in the middle.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

In a generator pipeline, when does a middle stage do its work?