Modules, packages and the standard library · about 40 min

Packages

Why this matters

One file per module is fine for a handful of modules, but real projects (and every library you will install) have dozens or hundreds. Packages let you group modules into directories with a hierarchy of names, such as email.mime.text or xml.etree.ElementTree. Understanding packages explains the dotted names you see in imports, what that mysterious __init__.py is for, why relative imports sometimes fail, what python3 -m really does, and where pip install puts things.

A package is a directory of modules

A package is a directory containing an __init__.py file. Its modules are called submodules:

project/
  main.py
  shop/
    __init__.py
    prices.py
    cart.py

# shop/prices.py
PRICES = {"apple": 0.5, "bread": 2.25}

def price_of(item):
    return PRICES[item]

# main.py
import shop.prices

print(shop.prices.price_of("bread"))

2.25

What happened when import shop.prices ran:

Python found the directory shop on sys.path (main.py's directory) with an __init__.py in it.
It created a module object for the package shop and executed shop/__init__.py in it. Packages are modules too; their code is the __init__.py file.
It then found prices.py inside the package's directory (packages have a __path__ attribute listing where to look for submodules), created the module shop.prices, and ran it.
It set the attribute prices on the shop module object to the submodule, stored both in sys.modules (as "shop" and "shop.prices"), and bound the name shop in main.

So shop.prices.price_of is a chain of attribute lookups: module shop, its attribute prices (a module), its attribute price_of (a function).

The other import forms work as you would expect:

from shop import prices            # binds prices -> the submodule
from shop.prices import price_of   # binds price_of -> the function
import shop.prices as sp           # binds sp -> the submodule

Note: import shop alone runs only shop/__init__.py. It does not automatically import the submodules, so shop.prices will raise AttributeError unless something (you, or __init__.py) imported it.

Packages nest: shop/payments/__init__.py makes shop.payments a subpackage. (A directory without __init__.py can also be imported, as a "namespace package", a feature meant for splitting one package across several directories. Always include __init__.py in your own packages; it makes the intent explicit.)

What goes in init.py

Often nothing at all: an empty file, or just a docstring. But because it runs on import shop, it is the place to define the package's public API, so users do not need to know your internal file layout:

# shop/__init__.py
"""A tiny shop: prices and shopping carts."""

from .prices import price_of
from .cart import Cart

__all__ = ["price_of", "Cart"]

Now from shop import price_of works, and you are free to move price_of to another submodule later without breaking users. __all__ lists the public names: it controls what from shop import * imports and tells readers and tools which names are the supported API. Names starting with _ are, by convention, private.

Absolute and relative imports

Inside a package, a module can import its siblings two ways:

# shop/cart.py
from shop.prices import price_of     # absolute: full path from a sys.path root
from .prices import price_of         # relative: "prices, in my own package"
from . import prices                 # the sibling module itself
from ..utils import money            # one package up (only inside nested packages)

A leading dot means "relative to the package this module belongs to"; each extra dot goes up one level. Relative imports keep working if you rename the top-level package, and make it obvious that a module is internal. Absolute imports are clearer for readers from outside. Both are fine; PEP 8 prefers absolute imports but accepts explicit relative ones inside packages.

Relative imports only work in a module that knows which package it is in. A module learns that from how it was imported: shop.cart has the package shop. That leads to the most common package error.

Running modules inside packages: python3 -m

If you run a submodule directly as a file:

$ python3 shop/cart.py
Traceback (most recent call last):
  File "/home/ada/project/shop/cart.py", line 1, in <module>
    from .prices import price_of
ImportError: attempted relative import with no known parent package

the file is executed as the top-level module __main__, with no package, so a leading dot means nothing. Worse, sys.path[0] is now project/shop, so even the absolute import shop would fail.

The fix is the -m switch: run a module by its import name.

$ python3 -m shop.cart

Python puts the current directory on sys.path, imports the package shop normally, and then runs shop/cart.py as __main__ while remembering that it belongs to shop. Relative and absolute imports both work. You have already used -m without thinking about it: python3 -m venv, python3 -m pip, python3 -m unittest, python3 -m json.tool.

If you run a package with -m, Python executes its __main__.py:

textkit/
  __init__.py
  __main__.py     <- run by: python3 -m textkit
  core.py

That is how a package becomes a command-line tool.

Third-party packages: site-packages, pip and venv

Packages you did not write live in site-packages directories, which are on sys.path after the standard library. pip is the installer: pip install requests downloads a distribution from the Python Package Index (PyPI) and unpacks its packages into a site-packages directory. Nothing more magical than that: afterwards import requests finds it on sys.path like any other package.

Installing everything into the one system Python causes conflicts: project A needs version 1 of a library, project B needs version 2, and your operating system's own Python tools need whatever they were tested with. (Debian and Fedora now refuse pip install into the system Python for this reason, with an "externally-managed-environment" error.) The answer is a virtual environment:

$ python3 -m venv .venv          # create one in the project directory
$ . .venv/bin/activate           # put .venv/bin first on PATH (optional)
(.venv) $ pip install requests   # installs into .venv/lib/python3.13/site-packages
(.venv) $ python -c "import sys; print(sys.prefix)"
/home/ada/project/.venv

.venv/
  bin/python      -> link to the real python3.13
  bin/pip
  bin/activate
  lib/python3.13/site-packages/   <- this project's third-party packages
  pyvenv.cfg      <- tells the interpreter "I am a venv, home is /usr/bin"

When .venv/bin/python starts, it sees pyvenv.cfg next to it and uses the venv's site-packages instead of the system one. One venv per project, never committed to git (it is easily rebuilt from a requirements.txt or pyproject.toml). The learnprog sandbox has no network, so you will not install anything here; the standard library already contains a great deal, as the next two lessons show.

Common mistakes

Running a package module as a file (python3 shop/cart.py) and getting attempted relative import with no known parent package. Use python3 -m shop.cart from the project root.
Expecting import shop to load submodules. Import them explicitly, or import them in __init__.py.
Naming a package after a standard module (email/, test/, logging/): the same shadowing problem as single files.
Circular imports through __init__.py. If __init__.py imports cart, and cart does from shop import price_of, then cart runs while shop/__init__.py is only half done. Import from the submodule directly (from .prices import price_of) inside the package.
Installing with the system pip. Make a venv.

Recap

A package is a directory with __init__.py; its modules are submodules named with dots, like shop.prices.
__init__.py runs on import of the package and is the place for its public API and __all__.
Relative imports (from .prices import x) work only in modules imported as part of a package.
python3 -m pkg.mod runs a module by import name; python3 -m pkg runs pkg/__main__.py.
pip installs packages into site-packages; a venv gives each project its own site-packages.

Going further: run python3 -c "import email, json; print(email.__path__, json.__file__)" to see how standard packages are laid out, and read the Python tutorial chapter "Modules", section 6.4 "Packages".

Exercises

Exercise 1 · quiz

Running a module inside a package

In the project below, shop/cart.py begins with from .prices import price_of.

project/
  shop/
    __init__.py
    prices.py
    cart.py

From the project directory, which command runs cart.py so that its relative import works?

python3 shop/cart.py
python3 -m shop.cart
python3 -m shop/cart.py
cd shop && python3 cart.py

Answer

python3 -m shop.cart

python3 shop/cart.py runs the file as a top-level script: its module has no parent package, so from .prices import ... fails with ImportError: attempted relative import with no known parent package, and sys.path[0] becomes shop/ rather than project/. python3 -m shop.cart imports shop first, then runs shop.cart as __main__ with its package set, and sys.path[0] is the current directory, so both absolute and relative imports resolve.

Exercise 2 · write code

Give a package a clean public face

The package shapes has two submodules already written:

shapes/circle.py defines area(radius) and perimeter(radius)
shapes/square.py defines area(side) and perimeter(side)

Users currently have to write from shapes.circle import area as circle_area. Write shapes/__init__.py so that this works instead:

import shapes
shapes.circle_area(1)      # area of a circle with radius 1
shapes.square_area(3)      # 9
shapes.circle.perimeter(1) # submodules are still reachable as attributes

Requirements:

Use relative imports (from .circle import ...).
Expose exactly the names circle_area and square_area as the package's public API by setting __all__ = ["circle_area", "square_area"].
Give the package a docstring.

File: shapes/__init__.py

Starting point

# shapes/__init__.py runs when someone does `import shapes`.

Hint 1

from .circle import area as circle_area imports the submodule and binds a new name in the package namespace.

Hint 2

Importing shapes.circle (even via a relative from-import) also sets the attribute shapes.circle on the package.

Hint 3

The docstring is a string literal as the very first statement.

Reference solution

"""Area and perimeter formulas for simple shapes."""

from .circle import area as circle_area
from .square import area as square_area

__all__ = ["circle_area", "square_area"]

Exercise 3 · write code

Make a package runnable with -m

The package textkit contains textkit/__init__.py (empty) and textkit/core.py, which defines:

shout(text): returns the text uppercased with ! appended
count_words(text): returns the number of whitespace-separated words

Write textkit/__main__.py so that python3 -m textkit reads all of stdin, and prints two lines:

words: <number of words>
<the whole input stripped of surrounding whitespace, shouted>

For the input hello there the output is:

words: 2
HELLO THERE!

Import from core with a relative import, and put the work in a main() function called under the __main__ guard. Read all of stdin with sys.stdin.read().

File: textkit/__main__.py

Starting point

import sys

Hint 1

python3 -m textkit runs textkit/__main__.py as the module textkit.__main__, so relative imports work.

Hint 2

from .core import count_words, shout

Hint 3

Strip the input before shouting so the trailing newline does not end up before the !.

Reference solution

"""Command-line entry point: python3 -m textkit"""
import sys

from .core import count_words, shout


def main():
    text = sys.stdin.read().strip()
    print("words:", count_words(text))
    print(shout(text))


if __name__ == "__main__":
    main()

Exercise 4 · quiz

What a virtual environment is

Which statement about virtual environments (python3 -m venv .venv) is correct?

A venv is a lightweight virtual machine that isolates the operating system
A venv is a directory with its own site-packages (and a python link), so packages installed with its pip do not affect the system Python or other projects
A venv makes Python code run faster by precompiling it
A venv copies the whole standard library so the project works without Python installed

Answer

A venv is a directory with its own site-packages (and a python link), so packages installed with its pip do not affect the system Python or other projects

A virtual environment is just a directory: bin/python (a link to the real interpreter), bin/pip, bin/activate, and its own lib/python3.13/site-packages. Running its python puts that site-packages on sys.path instead of the system one. Each project gets its own set of third-party packages and versions; the standard library and interpreter are shared.