Iterators & Generators
A for loop in Python does not "walk a collection" and knows nothing about length or indices. It runs exactly one script — it calls iter(obj) once, receives an iterator, and then calls next() repeatedly until StopIteration arrives, which it silently swallows. Everything iterable in Python — list, dict, a file, zip, range, a generator — is plugged into that single protocol, and your own data structure becomes usable in for the moment it implements two dunder methods.
Two things follow from the protocol, and interview questions are built around both. The first is the difference between an iterable and an iterator. An iterable hands out a fresh cursor on request, which is why a list can be traversed any number of times; an iterator is the cursor — it holds a position and is exhausted irreversibly. The second is generators, the shortest way to write an iterator: a function with yield in its body returns a generator object without executing a single line, and yield freezes the whole frame along with every local variable. Three classic traps grow from there — a generator is read once and yields nothing on a second pass, (x for x in y) does not produce a tuple at all, and a StopIteration raised inside a generator body turns into a RuntimeError. Work through the mechanics layer by layer.
Topic map
- Iterables — what makes an object usable in
for, and why alistis traversable many times while a cursor is traversable once. - The iterator protocol —
__next__,__iter__returningself,StopIteration, and howfordesugars intoiterplus repeatednext. - Writing an iterator class — a hand-written cursor, separating the container from the cursor, and the cost of "
__iter__returnsself". - Generator functions —
yieldin the body turns a function into a generator factory; calling it runs no code, and exhaustion is permanent. - yield and frame suspension — how the frame is paused, what happens to locals between resumes, and what
yield fromdelegates. - send, throw and close — the generator's two-way channel,
GeneratorExit, and honest resource finalization. - Comprehensions and generator expressions —
[...]versus(...), the non-existent "tuple comprehension", and the loop variable's scope. - Laziness and memory — O(1) versus O(n), streams larger than RAM, infinite sequences, and when a list is still the better choice.
Common mistakes and traps
| Mistake | Consequence |
|---|---|
| Confusing an iterable with an iterator | Expecting a repeatable traversal where a one-shot cursor lives — the second pass silently returns nothing |
Implementing __next__ without __iter__ | iter() fails with TypeError "object is not iterable" even though next() works on the object |
Not raising StopIteration in __next__ | The for loop never terminates — traversal becomes infinite |
| Expecting a generator to restart on a second pass | The second list(gen) returns [] — a second pass needs a new generator or a materialized list |
Treating (x for x in y) as a tuple | It is a generator expression — no len, no indexing; a tuple only comes from tuple(...) |
Calling send(x) before priming the generator | TypeError about sending a non-None value to a just-started generator |
Raising StopIteration inside a generator body | PEP 479 replaces it with a RuntimeError — instead of quiet termination you get a crash |
Returning self from a container's __iter__ | The container becomes one-shot — a nested loop over it yields a truncated result instead of a Cartesian product |
What interviews check
The topic is part of the mandatory minimum for a middle developer, and it almost always opens with definitions. You are asked how an iterable differs from an iterator — the correct answer is "an iterable hands out a fresh iterator from __iter__, an iterator implements __next__ and holds a position", and it closes half the follow-ups by itself. Then the consequences are probed — why an iterator's __iter__ returns self, and what next() does after the first StopIteration. Code follows: write your own reversed as a generator and as a class, or flatten a nested list with yield from. What is being watched there is not the algorithm but the protocol — is there an __iter__, is StopIteration raised, is a copy quietly materialized.
The second half of the conversation is about generators and laziness. The standard set — what calling a generator function returns, what list(gen) prints twice, how (...) differs from [...], and why a generator saves memory; the phrase "O(n) versus O(1)" is expected verbatim. At senior level, send/throw/close are added with a mandatory mention of priming and GeneratorExit, plus the semantics of yield from including the subgenerator's return value, and the iter() fallback to __getitem__. The typical mistake at every level is the same — the candidate describes the behaviour ("it walks the elements") but cannot name the calls it is assembled from.