Python Mastery · Chapter 8 · ~20 min · Intermediate

Generators & Iterators: Streaming Big Data

How do you process a 50 GB log file on a computer with only 8 GB of RAM? Meet Python generators: lazy evaluation, yield, and zero-memory streaming pipelines.

Know why lazy beats eager for big data
Understand yield as a pause button
Build streaming generator pipelines
Use iter() and next() confidently

1The RAM Crisis: Eager Lists vs. Lazy Streams

Standard Python collections (lists, dicts, sets) are eager: every single item must be computed and stored in physical RAM at once. If you load 10 million rows from a database into a list, your program will crash with MemoryError.

A generator is lazy: it calculates each value on demand, hands it to you, and immediately discards it before generating the next one.

Live Memory Simulator: List vs. Generator Expression

Notice how the list grows with dataset size (O(N)), while the generator stays at a flat ~112 bytes (O(1)).

[x * 2 for x in range(1,000,000)]Eager List

8.4 MB

Allocates all elements in RAM before executing the next line of code.

(x * 2 for x in range(1,000,000))Lazy Gen

112 bytes

Zero elements stored! Only remembers formula and pointer to current step.

Rule of thumb: If you only need to read through items in order once, never build a list. Use a generator!

2The Conveyor Belt: yield vs return

A normal function has a return statement: it gives you an answer, destroys its stack frame, and forgets all its local variables.

A function with yield is a generator factory. When called, it doesn't run the code immediately. Instead, each call to next() runs until it hits yield, pauses, and freezes its state.

Interactive Generator Stepper: yield vs return

Watch the function pause mid-execution and remember its exact line number.

# Generator Function Definition

def countdown_batches():

print("Start generator")

yield 10 # ⏸ Pauses here at line 3!

yield 20 # ⏸ Resumes, then pauses at line 4!

yield 30 # ⏸ Resumes, then pauses at line 5!

# Function ends: raises StopIteration!

Conveyor Belt (Emitted on Demand)

Conveyor belt empty. Press "next(gen)" to request item.

Generator instantiated: gen = countdown_batches(). Zero code has run yet!

Mental model

yield is a pause button on a cassette tape

return ejects the tape — the function is finished and its local variables are gone. yield presses pause: it hands out one value and remembers exactly where it stopped. Calling next() presses play again.

What happens to a generator's local variables after yield?

3Generator Expressions: () vs []

Just like list comprehensions, Python gives you generator expressions. The only difference is swapping square brackets for parentheses:

[x * 2 for x in data]

Builds the entire list in RAM immediately.

(x * 2 for x in data)

Creates a lightweight generator object that yields values one at a time.

The #1 Gotcha: Generators Are Single-Use (Exhaustion)

Unlike a list, once a generator yields all its items, it is empty forever. Looping over it again yields nothing!

gen = (x ** 2 for x in [1, 2, 3])

# Pass 1:
for n in gen:
    print(n)  # 1, 4, 9

# Pass 2:
for n in gen:
    print(n)  # NOTHING! Generator is exhausted!

Interactive Test:

Generator has 3 items waiting: [1, 4, 9]

Which uses almost no memory for a billion numbers?

4Building Streaming Big Data Pipelines

The true superpower of generators is pipeline chaining. You can connect multiple generator functions like Unix pipes (cat | grep | cut), processing gigabytes of data in a continuous stream without storing intermediate arrays.

Big Data Pipeline: Chained Generators

read_lines() ➔ filter_errors() ➔ emit_alerts()

Click "Stream Next Line" to watch log entries flow through the generator pipeline one-by-one.

5The Iterator Protocol: iter() and next()

Ever wonder how for item in items: actually works in Python? Behind the scenes, it executes this two-step protocol:

# What Python does behind every for-loop:
iterator = iter(collection)    # Calls __iter__()
while True:
    try:
        item = next(iterator)   # Calls __next__()
        print(item)
    except StopIteration:       # Loop finishes cleanly!
        break

6Challenge: Streaming Architect

Test your mastery of lazy evaluation, yield, generator expressions, and memory optimization. Score 5/5 to complete Chapter 8!

Generators & Iterators: Streaming Big Data · code lab

Edit the example, run it with Python, or step through the lines.

Python

def squares(limit):
    for number in range(limit):
        yield number * number

for value in squares(4):
    print(value)

Line-by-line view

1def squares(limit):
2    for number in range(limit):
3        yield number * number
4 
5for value in squares(4):
6    print(value)

Names

Run the example to inspect local names.

Objects in memory

Memory snapshots appear as Python runs.

Python output

Run the code to see its output.

1.What does the 'yield' keyword do inside a function?

2.How much memory does a generator expression (x for x in range(10_000_000)) take in Python?

3.What exception is raised when a generator runs out of items?

4.What happens if you loop over the exact same generator object twice?

5.How do you distinguish a list comprehension from a generator expression syntactically?

Run and benchmark the 10,000-sensor streaming generator in the Python Playground (code preloaded).

7Recap

Eager vs Lazy: Lists store all data in RAM; generators compute items on the fly (O(1) memory).
yield: Pauses execution, emits a value, and remembers the local frame for the next call.
Generator Expressions: Use (x for x in data) instead of [x for x in data] for zero-overhead loops.
Single-Use Exhaustion: A generator can only be iterated over once.
Pipeline Architecture: Chain generator stages to process endless streams and huge files cleanly.

Finish the challenge above to complete the chapter, or