The iterator protocol

Why this matters

for works on lists, strings, dicts, files, ranges, zip and enumerate objects, and on things that never end, like a stream of network messages. How can one statement handle all of them? Through a tiny agreement called the iterator protocol: two methods, __iter__ and __next__, and one exception, StopIteration. Once you know it, you can make your own objects work with for, list(), sum(), zip() and every other tool that consumes iterables, and you will understand why some objects can be looped over only once.

Iterables and iterators

You met iter() and next() in the iteration module. Now precisely:

• An iterable is any object that can produce an iterator: it has __iter__ returning one. Lists, tuples, strings, dicts, sets, files, ranges.
• An iterator is an object that produces values one at a time: it has __next__, which returns the next value or raises StopIteration when there are none left. An iterator also has __iter__, which returns itself, so an iterator is also an iterable.

nums = [10, 20]
it = iter(nums)        # calls nums.__iter__()
print(it)
print(next(it))        # calls it.__next__()
print(next(it))
print(next(it))

<list_iterator object at 0x7f...>
10
20
Traceback (most recent call last):
  ...
StopIteration

The list does not move; the separate list_iterator object holds a position (an index into the list). That separation is the key design idea:

  iterable (the list)                iterator (list_iterator)
  [10, 20, 30]   <---- refers to ----  position = 1
                       iter(nums)          next() returns nums[1], position becomes 2

What for really does

This loop:

for x in [10, 20, 30]:
    print(x)

is equivalent to:

_it = iter([10, 20, 30])
while True:
    try:
        x = next(_it)
    except StopIteration:
        break
    print(x)

The bytecode shows it directly: GET_ITER calls iter() once, and FOR_ITER calls __next__ each time round, jumping out of the loop when the iterator is exhausted. (CPython handles the "exhausted" signal from built-in iterators without actually creating an exception object, which keeps loops fast.) list(x), sum(x), sorted(x), x in y for many types, tuple unpacking a, b = x, and str.join(x) all use the same protocol.

Writing an iterator class

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for n in Countdown(3):
    print(n)
print(list(Countdown(5)))

3
2
1
[5, 4, 3, 2, 1]

That is the whole protocol. Any object with these two methods works everywhere an iterable is expected.

Exhaustion: the one-shot problem

Iterators have state, and once StopIteration has been raised, a well-behaved iterator keeps raising it. So an iterator can be consumed only once:

c = Countdown(3)
print(list(c))
print(list(c))

[3, 2, 1]
[]

No error, just nothing. This silent emptiness is a common bug source. Many built-ins return iterators, not lists: zip, map, filter, enumerate, reversed, and file objects:

pairs = zip("ab", [1, 2])
print(dict(pairs))
print(list(pairs))      # already used up

{'a': 1, 'b': 2}
[]

If you need the values more than once, store them: pairs = list(zip(...)).

Iterable versus iterator in your own classes

A list can be looped over any number of times, even in nested loops, because each for calls iter(the_list) and gets a fresh iterator. Your own collections should do the same: make the collection an iterable whose __iter__ returns a new iterator object each time, and keep the moving position in that iterator.

class Word:
    def __init__(self, text):
        self.text = text
    def __iter__(self):
        return WordIterator(self.text)

class WordIterator:
    def __init__(self, text):
        self.text, self.i = text, 0
    def __iter__(self):
        return self
    def __next__(self):
        if self.i >= len(self.text):
            raise StopIteration
        ch = self.text[self.i]
        self.i += 1
        return ch

w = Word("hi")
print([a + b for a in w for b in w])

['hh', 'hi', 'ih', 'ii']

The nested comprehension gets two independent iterators. If Word were its own iterator, the inner loop would exhaust it and the result would be wrong. In practice, the easiest way to write __iter__ is as a generator, or to return iter(self._items) for an inner collection, as you did with Playlist in the OOP module. The next lesson shows generators.

Infinite and lazy iterators

Nothing says an iterator must end. itertools.count() yields 0, 1, 2, ... forever. Iterators compute each value only when asked, so an infinite one costs nothing until you pull from it. Combine with something that stops:

import itertools
for n in itertools.count(10, 5):
    if n > 25:
        break
    print(n, end=" ")
print()
print(list(itertools.islice(itertools.count(), 4)))

10 15 20 25
[0, 1, 2, 3]

itertools.islice(it, n) takes just the first n items. Calling list() directly on an infinite iterator would run until memory runs out.

Common mistakes

• Reusing an exhausted iterator: the second loop silently does nothing. Convert to a list if you need several passes.
• __iter__ that returns something that is not an iterator: TypeError: iter() returned non-iterator of type 'list'. Return iter(self.items), not self.items.
• Calling next() on an iterable instead of an iterator: next([1, 2]) gives TypeError: 'list' object is not an iterator. Call iter() first.
• Forgetting to advance the state in __next__: the loop never ends and the check times out.
• Making a collection its own iterator: nested loops and second passes break.

Recap

• Iterable: has __iter__ returning an iterator. Iterator: has __next__ (value or StopIteration) and __iter__ returning itself.
• for x in obj calls iter(obj) once, then next() until StopIteration.
• Iterators are stateful and one-shot; zip, map, filter, enumerate and files return iterators.
• Containers should return a fresh iterator from each __iter__ call.

Going further: dis.dis("for x in y: pass") shows GET_ITER and FOR_ITER. Read the "Iterator Types" section of the standard library docs (under "Built-in Types").