Iteration · about 40 min

Built-ins that consume iterables

Why this matters

Many loops exist only to boil a sequence down to one answer: the total, the largest, "is any of these invalid?", "are all of these done?". Python ships built-in functions for these questions, written in C, that work on any iterable: lists, generators, files, zip objects. Using them makes code shorter and its intent obvious (if any(line.startswith("ERROR") for line in log) reads like the requirement it implements). Two more built-ins, map and filter, transform streams lazily. Knowing which of these consume eagerly and which are lazy is the key to using them correctly on large or endless data.

sum

sum(iterable, start=0) adds up the items:

>>> sum([4, 8, 15])
27
>>> sum(range(101))
5050
>>> sum(len(w) for w in ["a", "bb", "ccc"])
6
>>> sum([0.1] * 10)
0.9999999999999999

The last line is ordinary float rounding error accumulating over ten additions. math.fsum tracks the lost bits and gives 1.0. sum of an empty iterable is start, normally 0. Do not use sum to join strings (it refuses with a TypeError that tells you to use "".join(...), which is far more efficient).

min and max

min and max accept either one iterable or several separate arguments:

>>> max([3, 9, 2])
9
>>> max(3, 9, 2)
9
>>> min("python")
'h'

Like sorted, they accept key=: compare by the key, return the original item. On ties they return the first such item.

>>> words = ["pear", "fig", "banana"]
>>> min(words, key=len)
'fig'
>>> prices = {"tea": 3.5, "coffee": 2.8, "juice": 4.1}
>>> min(prices, key=prices.get)
'coffee'
>>> max(prices.items(), key=lambda kv: kv[1])
('juice', 4.1)

prices.get is a method used as a key function: for each key it returns that key's value.

An empty iterable has no minimum:

>>> max([])
ValueError: max() iterable argument is empty
>>> max([], default=None)
>>> max([], default=0)
0

default= (only with the single-iterable form) supplies a value for the empty case.

any and all

any(iterable) is True if at least one item is truthy. all(iterable) is True if every item is truthy. Both short-circuit: they stop iterating as soon as the answer is known.

>>> any([0, "", None, 5])
True
>>> all([1, "x", [0]])
True
>>> nums = [2, 4, 6, 7]
>>> all(n % 2 == 0 for n in nums)
False
>>> any(n > 100 for n in nums)
False

Short-circuiting with a generator expression means that with any(expensive(x) for x in items), expensive is called only until the first truthy result. Conceptually, any is:

def my_any(iterable):
    for item in iterable:
        if item:
            return True      # stop right here
    return False

and all returns False at the first falsy item. Note the empty cases: all([]) is True and any([]) is False, because there is no counterexample to "all are truthy" and no example of "one is truthy".

Pass a generator expression rather than a list comprehension: any([check(x) for x in items]) computes every check first, building a list, and loses the short-circuit.

map and filter

map(func, iterable) yields func(item) for each item. filter(func, iterable) yields the items for which func(item) is truthy.

>>> list(map(int, ["3", "14", "15"]))
[3, 14, 15]
>>> list(map(str.upper, ["a", "b"]))
['A', 'B']
>>> list(filter(lambda n: n > 0, [3, -1, 4, -5]))
[3, 4]
>>> list(filter(None, [0, 1, "", "x", None]))
[1, 'x']

filter(None, ...) keeps the truthy items. map can take several iterables, calling func with one item from each, stopping at the shortest, like zip:

>>> list(map(pow, [2, 3, 4], [5, 2, 1]))
[32, 9, 4]

The idiom map(int, line.split()) is the standard way to parse a line of numbers:

a, b, c = map(int, "10 20 30".split())

map and filter versus comprehensions

These pairs are equivalent:

map(f, xs)                  # (f(x) for x in xs)
filter(pred, xs)            # (x for x in xs if pred(x))

When you already have a named function (int, str.strip, len), map is concise. When you would need a lambda, a comprehension or generator expression is usually clearer: [x * 2 for x in xs] beats list(map(lambda x: x * 2, xs)).

Eager versus lazy

This is the important part. These built-ins fall into two groups:

Eager: consume the whole iterable, return a value	Lazy: return an iterator, compute on demand
`sum`, `min`, `max`, `sorted`, `len`*, `list`, `tuple`, `set`, `dict`, `"".join`	`map`, `filter`, `zip`, `enumerate`, `reversed`, generator expressions
`any`, `all` (eager, but stop early once the answer is known)

(*len does not iterate at all; it asks the container for its stored size, and works only on containers, not iterators.)

A lazy object does no work when created:

def noisy(n):
    print("computing", n)
    return n * 10

m = map(noisy, [1, 2, 3])      # prints nothing yet
print("created")
print(next(m))                 # computing 1, then 10
print(list(m))                 # computing 2, computing 3, then [20, 30]

created
computing 1
10
computing 2
computing 3
[20, 30]

Lazy pipelines let you chain steps without intermediate lists and process data that does not fit in memory, or that never ends. itertools.count(1) yields 1, 2, 3, ... forever; asking for the first big one is fine, because only the needed values are computed:

import itertools
evens_squared = map(lambda n: n * n, filter(lambda n: n % 2 == 0, itertools.count(1)))
print(next(evens_squared), next(evens_squared))   # 4 16

Calling list() or sum() on that infinite pipeline would never finish. And as always, a lazy iterator is one-shot: after list(m), m is empty.

Common mistakes

Expecting map to return a list. print(map(int, ["1"])) shows <map object at 0x...>. Wrap in list() when you need a list, or just iterate it.

Consuming twice. nums = map(int, words), then sum(nums) and max(nums): the second call sees an empty iterator and raises ValueError: max() iterable argument is empty. Make a list first.

min/max of nothing. Guard with if items: or pass default=.

Losing the short-circuit. any([slow(x) for x in xs]) evaluates everything. Drop the brackets.

sum of strings. sum(["a", "b"], "") raises TypeError: sum() can't sum strings [use ''.join(seq) instead].

Recap

sum, min, max reduce an iterable to one value; min/max take key= and default=, and return the first item on ties.
any and all short-circuit; all([]) is True, any([]) is False. Feed them generator expressions.
map(f, it) transforms and filter(pred, it) selects, lazily; comprehensions are often clearer when a lambda would be needed.
Eager functions consume the whole iterable; lazy ones compute on demand, work on infinite streams, and are one-shot.

Going further: browse the itertools module documentation (count, islice, takewhile, chain), which provides lazy building blocks that combine with everything in this lesson.

Exercises

Exercise 1 · quiz

any and all on nothing

What do all([]) and any([]) return? Pick the correct pair.

all([]) is True, any([]) is False
all([]) is False, any([]) is False
all([]) is True, any([]) is True
Both raise ValueError because the list is empty

Answer

all([]) is True, any([]) is False

all asks "is there any element that is falsy?"; with no elements, there is no counterexample, so it is True (the mathematical "vacuous truth"). any asks "is there at least one truthy element?"; there is none, so False. Remember this when validating possibly-empty input: all(valid(x) for x in rows) is True for zero rows.

Exercise 2 · quiz

max with a key

What does this print?

words = ["pear", "fig", "banana", "plum", "cherry"]
print(max(words, key=len))

Answer

max with key=len compares lengths, but returns the original element. Both banana and cherry have length 6; on ties max (and min) return the first maximal element encountered, so the answer is banana.

Exercise 3 · write code

Number statistics

Read one line of integers separated by spaces and print their count, sum, minimum, maximum and mean, in exactly this format (mean with 2 decimal places). For input 4 8 15 16 23 42:

count: 6
sum: 108
min: 4
max: 42
mean: 18.00

If the line is empty (no numbers), print exactly no numbers and nothing else. Use map to convert the words to integers.

File: main.py

Starting point

line = input()

Example

input

4 8 15 16 23 42

expected output

count: 6
sum: 108
min: 4
max: 42
mean: 18.00

Example

input

-3 -1 -2

expected output

count: 3
sum: -6
min: -3
max: -1
mean: -2.00

Hint 1

map(int, line.split()) converts every word; wrap it in list(...) because you need the numbers more than once.

Hint 2

min([]) raises ValueError, so check for an empty list first.

Hint 3

Format the mean with an f-string: f"mean: {value:.2f}".

Reference solution

nums = list(map(int, input().split()))
if not nums:
    print("no numbers")
else:
    print("count:", len(nums))
    print("sum:", sum(nums))
    print("min:", min(nums))
    print("max:", max(nums))
    print(f"mean: {sum(nums) / len(nums):.2f}")

Exercise 4 · write code

Validating with any and all

In solution.py, write check_password(pw) that returns a list of problems (strings), in this order, or an empty list if the password is fine:

"too short" if it has fewer than 8 characters
"no digit" if no character is a digit
"no uppercase" if no character is uppercase
"has space" if any character is a space

Use any(...) with generator expressions for the character checks, and the string methods isdigit() and isupper().

check_password("Secret123")      # []
check_password("short")          # ['too short', 'no digit', 'no uppercase']
check_password("Has space 1")    # ['has space']

Then write all_valid(passwords) that returns True if every password in the iterable has no problems (use all).

File: solution.py

Starting point

def check_password(pw):
    problems = []
    return problems

def all_valid(passwords):
    return False

Hint 1

any(c.isdigit() for c in pw) is True if at least one character is a digit.

Hint 2

An empty list is falsy, so not check_password(pw) means "no problems".

Reference solution

def check_password(pw):
    problems = []
    if len(pw) < 8:
        problems.append("too short")
    if not any(c.isdigit() for c in pw):
        problems.append("no digit")
    if not any(c.isupper() for c in pw):
        problems.append("no uppercase")
    if any(c == " " for c in pw):
        problems.append("has space")
    return problems

def all_valid(passwords):
    return all(not check_password(pw) for pw in passwords)

Exercise 5 · write code

Lazy search

Write first_over(numbers, limit) in solution.py that returns the first number in the iterable that is greater than limit, or None if there is none. Combine filter (or a generator expression) with next(iterator, None), so that the search stops at the first match.

The check passes an infinite iterator, so a solution that builds a list first will run forever (and time out):

import itertools
first_over(itertools.count(1), 10)   # 11   (count(1) yields 1, 2, 3, ... forever)
first_over([3, 1, 4], 5)             # None

File: solution.py

Starting point

def first_over(numbers, limit):
    big = [n for n in numbers if n > limit]
    return big[0] if big else None

Hint 1

filter(func, it) and (n for n in it if n > limit) are both lazy: nothing is computed until you ask.

Hint 2

next(lazy_iterator, None) asks for exactly one value, or returns None if there is none.

Reference solution

def first_over(numbers, limit):
    return next(filter(lambda n: n > limit, numbers), None)