Testing with unittest

Why this matters

Bare assert statements in a function work, but they stop at the first failure, give little detail, and you have to call every test function yourself. A test framework fixes all of that: it finds your tests automatically, runs each one in isolation so one failure does not hide the others, reports exactly what was expected and what was received, and returns an exit status that scripts and CI systems can act on. Python ships one in the standard library, unittest. (Many projects use the third-party pytest, which can also run unittest tests; the ideas carry over directly.)

The shape of a unittest file

Here is a module to test, textutils.py, with a bug in word_count:

def word_count(text):
    return len(text.split(" "))


def initials(name):
    return "".join(part[0].upper() for part in name.split())

And its tests, test_textutils.py:

import unittest

from textutils import initials, word_count


class TestWordCount(unittest.TestCase):
    def test_simple_sentence(self):
        self.assertEqual(word_count("the quick brown fox"), 4)

    def test_extra_spaces(self):
        self.assertEqual(word_count("  the   quick  "), 2)

    def test_empty(self):
        self.assertEqual(word_count(""), 0)


class TestInitials(unittest.TestCase):
    def test_two_names(self):
        self.assertEqual(initials("ada lovelace"), "AL")


if __name__ == "__main__":
    unittest.main()

This uses a class, which you will study properly in the object-oriented module. For now, treat it as a recipe:

• class TestWordCount(unittest.TestCase): declares a group of related tests. The name is up to you; starting it with Test is the convention. The part in parentheses says "this group is a unittest TestCase", which is what gives it the assert... methods.
• Each test is a function indented inside the class, whose name starts with test, and whose first parameter is self. Functions defined in a class are called methods.
• Inside a test, call checks as self.assertEqual(actual, expected) and friends. self is how a method reaches the tools of its TestCase.

Running the tests

From the directory containing both files:

$ python3 -m unittest test_textutils
..FF
======================================================================
FAIL: test_empty (test_textutils.TestWordCount.test_empty)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ada/proj/test_textutils.py", line 14, in test_empty
    self.assertEqual(word_count(""), 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 0

======================================================================
FAIL: test_extra_spaces (test_textutils.TestWordCount.test_extra_spaces)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ada/proj/test_textutils.py", line 11, in test_extra_spaces
    self.assertEqual(word_count("  the   quick  "), 2)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 8 != 2

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=2)

How to read it:

• The first line has one character per test: . pass, F fail, E error, s skipped.
• Each failure gets a block naming the test (module.Class.method), the traceback pointing at the assertion, and the comparison: AssertionError: 1 != 0 means actual 1, expected 0. "".split(" ") gives [''], one empty string, and " the quick ".split(" ") yields 8 pieces, most of them empty. The fix is text.split() with no argument, which splits on runs of whitespace and drops empty pieces.
• The summary line and the exit status (1 when anything failed, 0 when all passed) tell scripts and CI servers the verdict.

Useful ways to run:

$ python3 -m unittest                          # discover test*.py files in this directory tree
$ python3 -m unittest -v test_textutils         # verbose: one line per test with its name
$ python3 -m unittest test_textutils.TestWordCount.test_empty   # a single test
$ python3 test_textutils.py                     # works too, thanks to unittest.main()

Discovery (python3 -m unittest with no names) imports every file matching test*.py, finds every TestCase class in it, and runs every method whose name starts with test. That is why the naming matters: a method called check_empty is never run, and nothing warns you.

FAIL versus ERROR

• FAIL: an assertion ran and the values were wrong.
• ERROR: the test raised some other exception, like TypeError or IndexError, before it reached a verdict. Often the code under test crashed; sometimes the test itself is broken.

Every test runs independently: unittest catches whatever happens in one test, records it, and moves on. You see all the problems in one run instead of only the first.

The assert methods

Why not plain assert? Because the methods know what they compare and produce informative messages. assertEqual on two lists says which index differs; on two long strings it shows a diff. Every method also accepts a final msg argument for extra context.

Testing that errors are raised

def test_rejects_negative(self):
    with self.assertRaises(ValueError):
        sqrt_int(-4)

The with block passes if the code inside raises ValueError (or a subclass), and fails with AssertionError: ValueError not raised if it completes normally. This replaces the try/except/else dance from the previous lesson. To check the message too: with self.assertRaisesRegex(ValueError, "negative"):.

Many similar cases: subTest

def test_squares(self):
    for n, expected in [(0, 0), (3, 9), (-4, 16)]:
        with self.subTest(n=n):
            self.assertEqual(square(n), expected)

Without subTest, the first wrong case would end the test. With it, every case is checked and each failure is reported with its parameters (n=-4).

setUp: shared preparation

When several tests need the same starting data, put it in a method named setUp. unittest calls it before each test, giving every test a fresh copy:

class TestInventory(unittest.TestCase):
    def setUp(self):
        self.stock = {"apple": 3, "pear": 0}

    def test_remove_one(self):
        remove(self.stock, "apple")
        self.assertEqual(self.stock["apple"], 2)

    def test_remove_missing_raises(self):
        with self.assertRaises(KeyError):
            remove(self.stock, "kiwi")

self.stock stores the dictionary on the test object so the test method can reach it. Because setUp runs again before every test, a test that modifies self.stock cannot affect the next one. Tests must never depend on running in a particular order. tearDown is the matching cleanup method, run after each test even if it failed (useful for deleting temporary files).

What makes a good test suite

• Each test checks one behavior, and its name says which: test_strips_dashes_at_ends tells you what broke without reading the code.
• Tests are independent and repeatable: no shared mutable state, no dependence on the current time, network, randomness without a seed, or the order of a set.
• Tests are fast, so you run them constantly. A suite that takes ten minutes gets run once a day.
• Every test has been seen to fail. Break the code on purpose (or write the test before the fix) and confirm the test catches it. A test that has never failed might be testing nothing, as the misnamed method in the first exercise shows.

Common mistakes

• Test methods not starting with test: silently skipped. Watch the "Ran N tests" count.
• Forgetting self: def test_x(): inside the class gives TypeError: TestX.test_x() takes 0 positional arguments but 1 was given, reported as an ERROR.
• Calling assertEqual without self.: NameError: name 'assertEqual' is not defined.
• assertTrue(a, b) when you meant assertEqual(a, b): assertTrue treats b as the failure message and passes whenever a is truthy.
• Naming the test file after the module under test (slug.py tested by slug.py) or a standard module (unittest.py): shadowing again. Use the test_ prefix.
• Running from the wrong directory: ModuleNotFoundError: No module named 'textutils'. Run from the directory that contains the modules (it becomes sys.path[0] with -m).

Recap

• A unittest file defines class TestX(unittest.TestCase): with methods named test_... that take self and use self.assertEqual, self.assertRaises and friends.
• python3 -m unittest [-v] [module] discovers and runs tests, reports each failure with expected and actual values, and exits non-zero on failure.
• FAIL means a wrong value; ERROR means an unexpected exception.
• setUp builds fresh data before each test; subTest reports every case of a loop.

Going further: read the unittest documentation sections "Basic example" and "Organizing test code", and look at unittest.mock, which replaces parts of your system (files, clocks, network calls) with fakes during a test.