Testing and debugging · about 45 min

Thinking in tests and assert

Why this matters

So far you have checked your programs by running them and looking at the output. That works once. But programs change: you fix a bug, add a feature, clean up a function, and something that used to work quietly breaks. A test is code that checks other code automatically, so you can re-check everything in a second after every change. Professional programmers spend a large share of their time writing tests, not because they enjoy paperwork, but because tests are what let them change code without fear. learnprog itself checks your exercises with tests; in this module you write them.

What a test is

A test runs a piece of code with a known input and compares the actual result with the expected result:

   input  -->  code under test  -->  actual result
                                          |
                     expected result -- compare --> pass / fail

The smallest useful unit is a function, which is one reason the earlier modules pushed you toward functions that take arguments and return values instead of reading input and printing. A function like that is trivial to test; a script that mixes input(), logic and print() is not.

assert

Python's built-in tool for a check is the assert statement:

assert condition, "message shown if it fails"

If the condition is true, nothing happens. If it is false, Python raises AssertionError with the message:

def area(width, height):
    return width * height

assert area(3, 4) == 12
assert area(0, 5) == 0
assert area(2, 2) == 5, "2 by 2 should be 4, this assert is wrong on purpose"
print("all good")
Traceback (most recent call last):
  File "demo.py", line 6, in <module>
    assert area(2, 2) == 5, "2 by 2 should be 4, this assert is wrong on purpose"
           ^^^^^^^^^^^^^^^
AssertionError: 2 by 2 should be 4, this assert is wrong on purpose

The first failing assert stops the program, so all good never prints. The traceback shows which check failed. Underneath, assert cond, msg compiles to roughly if __debug__ and not cond: raise AssertionError(msg).

Grouping asserts into test functions

Collect the checks for one function into a test function, conventionally named test_<thing>:

def is_even(n):
    return n % 2 == 0


def test_is_even():
    assert is_even(0)
    assert is_even(2)
    assert not is_even(3)
    assert is_even(-4)
    assert not is_even(-1)


test_is_even()
print("test_is_even passed")

Each assert is a test case: one input with one expected outcome. Test functions take no arguments and return nothing: they either complete (pass) or raise AssertionError (fail). That convention is exactly what testing tools such as pytest look for, and what the next lesson's unittest formalizes.

Choosing test cases

The hard part of testing is not the syntax but deciding what to check. Good tests are chosen to expose likely bugs.

Typical cases confirm the ordinary path works: average([2, 4, 6]) == 4.

Edge cases sit at boundaries, where most bugs live:

Error cases check that bad input is rejected the way the function promises: the right exception, not a wrong answer. You check those with try/except:

def test_average_empty():
    try:
        average([])
    except ValueError:
        pass                      # the promised error: good
    else:
        assert False, "average([]) should raise ValueError"

The else branch of a try runs only if no exception occurred, which here means the function failed to complain.

Rules with exceptions deserve one case per branch. For the leap-year rule (divisible by 4, except centuries, except every 400 years) you need at least four cases: 2024, 2023, 1900 and 2000. A test suite that only checks 2024 passes even for the naive year % 4 == 0.

Here is a useful way to judge your tests: imagine a plausible bug, and ask whether some test would fail. "What if I forgot to sort?" Then you need an unsorted input. "What if I modified the caller's list?" Then you need a check that the argument is unchanged afterwards. This is exactly how the exercises in this lesson grade you: they plant bugs and see whether your tests notice. (Professionals call this idea mutation testing.)

Tests as specification and safety net

Tests do more than catch mistakes:

assert is not input validation

assert statements can be switched off. Running python3 -O program.py ("optimize") compiles the program with the built-in constant __debug__ set to False, and every assert is removed from the bytecode. So:

def set_age(age):
    assert age >= 0          # WRONG: disappears under -O
    ...

def set_age(age):
    if age < 0:              # RIGHT: always runs
        raise ValueError(f"age must be non-negative, got {age}")
    ...

Use assert in tests, and inside your own code only for internal sanity checks, conditions that can only be false if you made a programming mistake ("this list should be sorted by now"). Never use it to check user input, file contents or anything from outside.

Common mistakes

Recap

Going further: write a test-first version of a small function, for example clamp(value, low, high): write test_clamp with five asserts before writing clamp. Then read the assert section of the Python Language Reference ("Simple statements").

Exercises

Exercise 1 · quiz

When asserts vanish

A web form handler starts with:

def withdraw(account, amount):
    assert amount > 0, "amount must be positive"
    account.balance -= amount

The service is deployed with python3 -O server.py. A user submits amount = -500. What happens?

Answer

  • The assert is skipped, so the balance increases by 500

-O ("optimize") compiles with __debug__ set to False, and every assert statement is removed from the bytecode. The check never runs, the negative amount is subtracted, and the balance grows. Asserts are for catching programmer mistakes during development and testing, never for validating input or enforcing rules: for that, use if amount <= 0: raise ValueError(...).

Exercise 2 · write code

A function and its tests

In solution.py write two functions.

  1. is_leap(year) returns True if year is a leap year in the Gregorian calendar, else False. The rule: divisible by 4, except years divisible by 100, which are leap years only if also divisible by 400. (2024 yes, 2023 no, 1900 no, 2000 yes.)
  2. test_is_leap() contains assert statements that check is_leap thoroughly, and returns nothing.

The checker runs your test_is_leap() against your is_leap (it must pass), and then against several buggy versions of is_leap (each one must make your test fail with an AssertionError). So your tests must cover every part of the rule, including the century cases.

File: solution.py

Starting point

def is_leap(year):
    return year % 4 == 0


def test_is_leap():
    assert is_leap(2024)
Hint 1

Write one assert per part of the rule: a plain leap year, a plain non-leap year, a century, and a 400-year century.

Hint 2

Use assert not is_leap(1900) to check that something is False.

Hint 3

One possible rule in Python: year % 4 == 0 and (year % 100 != 0 or year % 400 == 0).

Reference solution
def is_leap(year):
    """Return True if year is a Gregorian leap year."""
    return year % 4 == 0 and (year % 100 != 0 or year % 400 == 0)


def test_is_leap():
    assert is_leap(2024), "ordinary leap year"
    assert not is_leap(2023), "ordinary common year"
    assert not is_leap(1900), "century not divisible by 400"
    assert is_leap(2000), "century divisible by 400"

Exercise 3 · quiz

Choosing edge cases

You are testing average(numbers), which should return the mean of a list of numbers and raise ValueError for an empty list. Which inputs are good edge cases to include? Select all that apply.

Answer

  • An empty list []

  • A list with a single element [7]

  • A list containing negative numbers and zero

  • A list whose mean is not a whole number, such as [1, 2]

Edge cases live at the boundaries where code tends to break: empty input (division by zero?), a single item (off-by-one?), negatives and zero (sign handling), and results that are not whole numbers (integer division by mistake?). Ten more ordinary lists exercise the same path as the first and add almost nothing. Good tests are chosen, not piled up.

Exercise 4 · write code

Tests that catch real bugs

The module stats.py (in your working directory) defines median(numbers): the middle value of the sorted numbers, or the mean of the two middle values when the count is even. It must not modify its argument, and it raises ValueError for an empty list.

Write test_median.py containing a function test_median() that uses assert statements (and a try/except for the empty case) to check median thoroughly. Import it with from stats import median.

The checker runs test_median() against the correct stats.py (must pass) and against five buggy versions (each must make your test raise AssertionError). Think about: odd and even lengths, unsorted input, whether the input list is left unchanged, and the empty list.

File: test_median.py

Starting point

from stats import median


def test_median():
    assert median([1, 2, 3]) == 2
Hint 1

Use unsorted inputs, e.g. [3, 1, 2], so a version that forgets to sort gets the wrong answer.

Hint 2

Check an even-length list whose two middle values differ: the answer should be their mean.

Hint 3

To check the argument is unchanged: keep a list in a variable, call median on it, then compare it with its original contents.

Hint 4

For the empty list: try: median([]) / except ValueError: pass / else: assert False, "should raise".

Reference solution
from stats import median


def test_median():
    assert median([5]) == 5, "single element"
    assert median([3, 1, 2]) == 2, "odd length, unsorted"
    assert median([4, 1, 3, 2]) == 2.5, "even length, unsorted"
    data = [3, 1, 2]
    median(data)
    assert data == [3, 1, 2], "median must not modify its argument"
    try:
        median([])
    except ValueError:
        pass
    else:
        assert False, "median([]) should raise ValueError"