Python: sharing common tests in unittest

Example code

In all the examples below, we’ll be testing these two classes which have a common interface:

class Armadillo:
    def speak(self) -> str:
        return "Hrrr!"


class Okapi:
    def speak(self) -> str:
        return "Gronk!"

With a nested base class

This approach defines the base class inside a plain container class:

from unittest import TestCase

from example import Armadillo, Okapi


class Common:
    class BaseAnimalTests(TestCase):
        animal_class: type  # To be defined in subclasses

        def test_speak(self):
            sound = self.animal_class().speak()
            self.assertIsInstance(sound, str)
            self.assertGreater(len(sound), 0)


class ArmadilloTests(Common.BaseAnimalTests):
    animal_class = Armadillo


class OkapiTests(Common.BaseAnimalTests):
    animal_class = Okapi

The plain class Common exists only as a namespace, and it can be called whatever you like. It works because unittest’s loader only collects TestCase subclasses that it finds as attributes of the module. BaseAnimalTests is an attribute of Common, so the loader never sees it, whilst the two module-level subclasses are collected as usual:

$ python -m unittest -v
test_speak (tests.ArmadilloTests.test_speak) ... ok
test_speak (tests.OkapiTests.test_speak) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

This is my preferred approach, compared to those covered below. The wrapper class may look strange at first, but at least it’s simple and contained, and both unittest and pytest correctly skip the base class.

You can extend this technique to sharing the base class between test modules. Move the container into another module and import it from there:

from example import Armadillo
from tests.common import Common


class ArmadilloTests(Common.BaseAnimalTests):
    animal_class = Armadillo

The costs of this approach are an extra level of indentation for the base class, and the Common. prefix at each point of use. I find those a small price to pay.

With a deleted base class

This approach, which this post previously recommended, defines the base class at module level, uses it, and then hides it with del:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests(TestCase):
    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests):
    animal_class = Okapi


del BaseAnimalTests  # Hide base class from test discovery

The del is needed to prevent unittest from collecting and running BaseAnimalTests itself. Without it, unittest would run it, and it would fail because it does not define the required animal_class attribute.

This runs the same two tests as the nested approach. But the del acts at a distance, sitting at the bottom of the file, potentially a long way from the class that it removes. Define a subclass after it and the module fails to import:

$ python -m unittest -v
tests (unittest.loader._FailedTest.tests) ... ERROR
...
    class PangolinTests(BaseAnimalTests):
                        ^^^^^^^^^^^^^^^
NameError: name 'BaseAnimalTests' is not defined

That’s a loud failure rather than a silent one, so it’s a papercut rather than a hazard. The bigger limitation is that del removes the name from the module that defines it, so this approach doesn’t extend to sharing the base class between test modules.

With a test class mixin

This approach puts the common tests in a mixin class that does not inherit from unittest.TestCase:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests:
    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests, TestCase):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests, TestCase):
    animal_class = Okapi

No del is needed here: unittest will not collect BaseAnimalTests because it does not inherit from unittest.TestCase. But this approach has at least a couple of drawbacks:

1. If any base class forgets to inherit from both BaseAnimalTests and unittest.TestCase, it will not be collected by unittest, and its tests will not run. This may be hard to notice. Even a defence like enforcing 100% coverage on your tests, as Ned Batchelder implores we use, may not help, since subclasses may not contain any tests of their own.

2. Type checkers will raise errors about undefined methods in the base class, for example:

   $ mypy --check-untyped-defs tests.py
   tests.py:11: error: "BaseAnimalTests" has no attribute "assertIsInstance"  [attr-defined]
   tests.py:12: error: "BaseAnimalTests" has no attribute "assertGreater"  [attr-defined]
   Found 2 errors in 1 file (checked 1 source file)
   

   I previously blogged about writing type-checked mixin classes, with the conclusion being that they should inherit from their intended base class. That means returning to either approach above, where the base class is a subclass of unittest.TestCase.

With pytest: using the __test__ attribute

If you use pytest to run your unittest classes, you can use a __test__ attribute to prevent collection of a specific class:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests(TestCase):
    __test__ = False  # Hide from test discovery

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.__test__ = True  # Enable test discovery for subclasses

    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests):
    animal_class = Okapi

__test__ controls whether pytest should collect a class, or not. pytest respects it as a lightly-documented feature, originally copied from the historical nose runner. The approach above hides the base class but exposes the subclasses, through some automatic configuration in __init_subclass__.

Pytest collects and runs only the subclasses, as expected:

$ pytest -v
===== test session starts ======
...
collected 2 items

test_example.py::ArmadilloTests::test_speak PASSED                                                                                                                                     [ 50%]
test_example.py::OkapiTests::test_speak PASSED                                                                                                                                         [100%]

====== 2 passed in 0.00s =======

This approach is a little bit more complex than the previous ones, and it only works with pytest. Still it is nice that pytest gives you that control. If you have a lot of common test classes, the technique could be wrapped up into a class decorator.

Fin

I think it would be neat if unittest gained some functionality here, like a TestCase decorator to prevent collection of a specific class but not its subclasses. Until that hypothetical future, though, I think nesting the base class is the way to go.

May your common tests work towards the common good,

—Adam