How for really works: iterables and iterators

Why this matters

You have used for on lists, strings, ranges and dicts since the control flow module, and it "just worked". Underneath, every one of those loops uses the same small protocol: ask the object for an iterator, then ask the iterator for the next value until it says it is finished. Knowing that protocol explains a family of puzzling behaviors (why a second loop over some objects prints nothing, why files and zip results can only be read once, why range(10**12) uses no memory) and it is the foundation for generators later in the track.

Iterables and iterators

Two words, two roles:

• An iterable is anything you can loop over: a list, a string, a tuple, a dict, a set, a range, a file. Formally: something you can pass to iter() to get an iterator.
• An iterator is an object that produces values one at a time and remembers where it is. You ask it for the next value with next(). When it has nothing left, it raises StopIteration.

Try it in the REPL:

>>> colors = ["red", "green", "blue"]
>>> it = iter(colors)
>>> next(it)
'red'
>>> next(it)
'green'
>>> next(it)
'blue'
>>> next(it)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

The list itself does not move or change. The iterator it is a separate, small object holding a reference to the list and a position:

 colors ──> list ["red", "green", "blue"]
                     ▲
 it ──> list_iterator(list=colors, index=3)   # after three next() calls

You can make as many independent iterators over the same list as you like; each has its own position.

What for desugars to

A for loop is shorthand. This loop:

for color in colors:
    print(color)

does essentially this:

it = iter(colors)              # 1. get an iterator from the iterable
while True:
    try:
        color = next(it)       # 2. ask for the next value
    except StopIteration:      # 3. no more values: leave the loop
        break
    print(color)               # 4. run the body with the name bound

The try/except lines catch an error instead of crashing; the exceptions module covers them in detail. For now, read it as "if next signals the end, break".

You can see the protocol in the bytecode too. dis.dis("for x in xs: pass") shows GET_ITER (call iter()), then a loop around FOR_ITER (call next(), and jump out of the loop on exhaustion). The interpreter does not have special cases for lists versus strings versus files: it only speaks this protocol, and each type implements it in its own way.

What different iterables give you

>>> list("hey")
['h', 'e', 'y']
>>> list({"a": 1, "b": 2})
['a', 'b']
>>> list(range(3))
[0, 1, 2]

list(...) itself just iterates its argument and collects the values, which makes it a handy way to see what an iterable produces.

Lazy iterables: range

range(1_000_000_000_000) does not create a trillion integers. A range object stores only its start, stop and step, and its iterator computes the next number when asked:

>>> import sys
>>> sys.getsizeof(range(10**12))
48
>>> sys.getsizeof(list(range(1000)))
8056

This is lazy evaluation: values are produced only when next is called. Many tools you will meet (zip, map, enumerate, file objects, generators) are lazy for the same reason: they let you process streams of data larger than memory, one item at a time.

One-shot iterators

Here is the most important practical consequence. A container like a list can be iterated many times, because each for loop calls iter() and gets a fresh iterator. An iterator, however, is its own iterator: calling iter() on it returns the same object, with the position where it left off.

>>> it = iter([1, 2, 3])
>>> iter(it) is it
True
>>> list(it)
[1, 2, 3]
>>> list(it)
[]

The second list(it) finds the iterator already exhausted. No error, just nothing. This trips people up with zip, map and files:

pairs = zip([1, 2], ["a", "b"])
print(list(pairs))   # [(1, 'a'), (2, 'b')]
print(list(pairs))   # []  -- already consumed

If you need the values twice, store them in a list first: pairs = list(zip(...)).

The default argument of next

next(it) raises StopIteration at the end. next(it, default) returns default instead, which is convenient when you want just the first item, or a fallback:

>>> next(iter([]), "empty")
'empty'
>>> words = ["alpha", "beta"]
>>> next(iter(words), "none")
'alpha'

Because a for loop and explicit next calls can share one iterator, you can also consume items "ahead" inside a loop, as the last exercise asks you to do:

lines = iter(["name:", "Ada", "name:", "Grace"])
for label in lines:
    value = next(lines)       # take the following line as well
    print(label, value)

name: Ada
name: Grace

Common mistakes

Looping over a number.

for i in 5:
    print(i)

TypeError: 'int' object is not iterable

You meant range(5).

Reusing a consumed iterator. A second loop over a zip, map, file or generator silently does nothing. Convert to a list if you need several passes.

Changing a list while iterating it. The list iterator tracks an index; removing items shifts later items down, so some are skipped:

nums = [1, 2, 2, 3]
for n in nums:
    if n == 2:
        nums.remove(n)
print(nums)   # [1, 2, 3]  -- one 2 survived

Iterate over a copy (for n in nums[:]) or build a new list. Changing a dict's size while iterating it is caught for you: RuntimeError: dictionary changed size during iteration.

Calling next on a list. next([1, 2]) raises TypeError: 'list' object is not an iterator. A list is iterable, not an iterator; call iter() first.

Recap

• An iterable can produce an iterator via iter(); an iterator produces values via next() and raises StopIteration when done.
• for x in thing is iter() once, then next() repeatedly until StopIteration.
• Containers give a fresh iterator each time; iterators are one-shot and, once exhausted, stay empty.
• Lazy iterables such as range compute values on demand and use constant memory.
• next(it, default) avoids the exception at the end.

Going further: in the REPL, run import dis; dis.dis("for x in xs: print(x)") and find GET_ITER and FOR_ITER. Then read "Iterator Types" in the Python documentation (Library reference, Built-in Types).