Under the hood: internals and concurrency · about 45 min

Processes and concurrent.futures

Why this matters

The last lesson showed that threads do not speed up CPU-heavy pure-Python code because of the GIL. Yet many real jobs are exactly that: crunching numbers, parsing thousands of files, resizing images, running simulations. The standard answer in Python is to use several processes, each with its own interpreter and its own GIL. This lesson covers how that works, what it costs, and the high-level concurrent.futures interface that makes both thread and process pools easy and interchangeable.

Processes versus threads

A process has its own memory space. Two processes running Python have two separate interpreters, two heaps of objects and two GILs, so the operating system can run them on two cores at the same time.

main process                    worker process 1        worker process 2
+-------------------+  pipe     +------------------+    +------------------+
| interpreter + GIL | <-------> | interpreter+GIL  |    | interpreter+GIL  |
| your objects      |  pickled  | its own objects  |    | its own objects  |
+-------------------+   bytes   +------------------+    +------------------+

The price of isolation:

Starting a process costs milliseconds and memory (a whole interpreter), far more than a thread.
Communication goes through pipes. Arguments and results are pickled (serialized to bytes with the pickle module), sent, and unpickled on the other side. Big arguments cost time to copy; some objects (open files, locks, lambdas) cannot be pickled at all.
No shared state by default: a worker that modifies a global changes its own copy. Results must be returned.

Rule of thumb: use processes for CPU-bound work made of reasonably big chunks with small inputs and outputs.

concurrent.futures

The concurrent.futures module gives one interface for both kinds of pools:

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

An executor owns a pool of workers. You hand it work and get back results.

map

executor.map(func, iterable, ...) is the parallel version of the built-in map:

from concurrent.futures import ProcessPoolExecutor

def square(n):
    return n * n

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=2) as executor:
        results = list(executor.map(square, range(6)))
    print(results)

[0, 1, 4, 9, 16, 25]

Key facts:

Results come back in input order, regardless of which task finishes first.
With several iterables, func gets one argument from each: executor.map(pow, [2, 3], [5, 2]) computes pow(2, 5) and pow(3, 2).
The with block shuts the pool down at the end, waiting for running tasks to finish.
If a task raises an exception, it is re-raised in the main process when you reach that result.
For many tiny tasks, pass chunksize= (process pools only) so each round trip carries a batch.

submit and futures

executor.submit(func, *args) schedules one call and immediately returns a Future: an object standing for a result that will exist later.

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def slow(n):
    time.sleep(0.1 * n)
    return n * 10

with ThreadPoolExecutor(max_workers=3) as executor:
    futures = {executor.submit(slow, n): n for n in [3, 1, 2]}
    for fut in as_completed(futures):
        print(futures[fut], "->", fut.result())

1 -> 10
2 -> 20
3 -> 30

fut.result() waits if necessary, then returns the value or re-raises the task's exception. as_completed yields futures in completion order, useful for progress reporting. Other methods: fut.done(), fut.cancel(), fut.exception().

Because both executors share the interface, switching between threads and processes is often a one-word change, which makes it easy to measure which works better.

The main guard and start methods

How does a worker process get your function? It depends on the start method:

fork (the default on Linux in Python 3.13): the OS duplicates the running process. The child starts with a copy of everything already imported and defined. Fast, but copying a process that has threads running can deadlock, so Python 3.14 changes the Linux default to forkserver.
spawn (default on macOS and Windows): a fresh interpreter starts and imports your main module to find the function by name.
forkserver: a clean helper process is started early and forks workers on request.

With spawn and forkserver, importing your main script in the worker runs its top-level code again. If that top-level code creates a pool, each worker would try to create a pool, and so on. Python detects this and raises an error. The cure is the guard you met in the modules lesson:

def work(x):            # top level: workers can import it
    ...

if __name__ == "__main__":
    # only runs in the original process, not when a worker imports this file
    with ProcessPoolExecutor() as ex:
        ...

Always write it, even on Linux, so your code works everywhere and on future Python versions. Also define worker functions at the top level of a module: functions are pickled by name (__main__.work), so lambdas and nested functions cannot be sent.

multiprocessing

concurrent.futures is built on the lower-level multiprocessing module, which mirrors the threading API and adds tools for when you need more control:

import multiprocessing as mp

def worker(q, n):
    q.put(n * n)

if __name__ == "__main__":
    q = mp.Queue()
    procs = [mp.Process(target=worker, args=(q, n)) for n in range(3)]
    for p in procs: p.start()
    for p in procs: p.join()
    print(sorted(q.get() for _ in range(3)))

[0, 1, 4]

mp.Queue pickles items through a pipe; mp.Pool is an older pool API; mp.Value, mp.Array and multiprocessing.shared_memory allow genuinely shared memory for simple data. Note sorted(...): results from a queue arrive in completion order, which varies.

How many workers?

ProcessPoolExecutor() defaults to the number of CPUs (os.process_cpu_count() in 3.13). More processes than cores just adds switching and memory overhead for CPU-bound work. Thread pools default to min(32, cpus + 4), since threads mostly wait. Measure with time.perf_counter() instead of guessing: for small jobs, the startup and pickling overhead of processes can make the parallel version slower than the plain loop.

Common mistakes

No __main__ guard: with the spawn method you get RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.
Lambdas or nested functions as tasks in a process pool: PicklingError: Can't pickle <function <lambda> ...>: attribute lookup <lambda> on __main__ failed.
Expecting globals to update: a worker's changes to a global variable happen in its copy. Return values instead.
Huge arguments: sending a 1 GB list to each task means pickling 1 GB each time. Send file names or indices and let workers load their own data.
Using processes for I/O-bound work: threads (or asyncio, next lesson) are cheaper.

Recap

Processes have separate memory and separate GILs, so they run CPU-bound Python in parallel, at the cost of startup time and pickling all data passed between them.
ThreadPoolExecutor and ProcessPoolExecutor share one API: map (results in input order), submit returning a Future, as_completed for completion order.
Guard pool-creating code with if __name__ == "__main__": and define task functions at module top level.
multiprocessing offers Process, Queue, Pool and shared memory for finer control.

Going further: read the "Programming guidelines" section of the multiprocessing docs, then time count_primes from the exercise over larger ranges with 1, 2 and 4 workers on your own machine.

Exercises

Exercise 1 · quiz

Threads or processes?

Select all jobs where a ProcessPoolExecutor is a better fit than a ThreadPoolExecutor on standard CPython. (Select all that apply.)

Resizing 500 images with a pure-Python pixel loop
Downloading 500 web pages
Counting primes in pure Python over large ranges
Waiting on 50 slow database queries

Answer

Resizing 500 images with a pure-Python pixel loop
Counting primes in pure Python over large ranges

Processes each have their own interpreter and their own GIL, so pure-Python CPU-bound work runs truly in parallel on several cores. Downloads and database queries are I/O-bound: threads already overlap the waiting, and they are cheaper to start and share memory without pickling.

Exercise 2 · quiz

Order of results

Task 0 takes 3 seconds, task 1 takes 1 second, task 2 takes 2 seconds. With a pool of 3 workers, in what order does list(executor.map(run, [0, 1, 2])) return the results? Type the task numbers separated by spaces.

Answer

executor.map yields results in the order of the inputs, whatever order the tasks finish in (it waits for result 0 before yielding it, even if 1 and 2 are already done). If you want results as soon as each finishes, submit tasks and iterate with concurrent.futures.as_completed.

Exercise 3 · write code

Count primes in parallel

The starter counts primes in four ranges one after another with count_primes(lo, hi). Change only the main part so that the four ranges are processed by a ProcessPoolExecutor(max_workers=2) using executor.map, and keep the output identical:

[0, 1000): 168
[1000, 2000): 135
[2000, 3000): 127
[3000, 4000): 120
total: 550

Keep the work inside an if __name__ == "__main__": block. (executor.map with two input lists calls count_primes(lo, hi) with one item from each list.)

File: main.py

Starting point

def is_prime(n):
    if n < 2:
        return False
    i = 2
    while i * i <= n:
        if n % i == 0:
            return False
        i += 1
    return True


def count_primes(lo, hi):
    return sum(1 for n in range(lo, hi) if is_prime(n))


los = [0, 1000, 2000, 3000]
his = [1000, 2000, 3000, 4000]
counts = [count_primes(lo, hi) for lo, hi in zip(los, his)]
for lo, hi, c in zip(los, his, counts):
    print(f"[{lo}, {hi}): {c}")
print("total:", sum(counts))

Example

expected output

[0, 1000): 168
[1000, 2000): 135
[2000, 3000): 127
[3000, 4000): 120
total: 550

Hint 1

Import it with from concurrent.futures import ProcessPoolExecutor.

Hint 2

with ProcessPoolExecutor(max_workers=2) as executor: then counts = list(executor.map(count_primes, los, his)).

Hint 3

Indent all the top-level work under if __name__ == "__main__":; leave the function definitions at the top level so worker processes can find them.

Reference solution

from concurrent.futures import ProcessPoolExecutor


def is_prime(n):
    if n < 2:
        return False
    i = 2
    while i * i <= n:
        if n % i == 0:
            return False
        i += 1
    return True


def count_primes(lo, hi):
    return sum(1 for n in range(lo, hi) if is_prime(n))


if __name__ == "__main__":
    los = [0, 1000, 2000, 3000]
    his = [1000, 2000, 3000, 4000]
    with ProcessPoolExecutor(max_workers=2) as executor:
        counts = list(executor.map(count_primes, los, his))
    for lo, hi, c in zip(los, his, counts):
        print(f"[{lo}, {hi}): {c}")
    print("total:", sum(counts))

Exercise 4 · quiz

Why did this fail?

from concurrent.futures import ProcessPoolExecutor

if __name__ == "__main__":
    with ProcessPoolExecutor() as ex:
        print(list(ex.map(lambda x: x * 2, [1, 2, 3])))

This fails with an error mentioning pickle, while the same code with ThreadPoolExecutor prints [2, 4, 6]. Why?

Lambdas are not allowed in list comprehensions
Work sent to another process must be pickled, and functions are pickled by their importable name; a lambda has none
ProcessPoolExecutor only accepts built-in functions
The lambda captures the GIL

Answer

Work sent to another process must be pickled, and functions are pickled by their importable name; a lambda has none

Processes do not share memory. The executor serializes (pickles) the function and its arguments, sends the bytes through a pipe, and the worker unpickles them. Functions are pickled as a reference like __main__.double, which the worker looks up by name. A lambda, or a function defined inside another function, cannot be found that way, so pickling fails. Define worker functions at the top level of a module. Threads share memory, so nothing is pickled.