New Testing Features in Django 4.0

1. Random test order with --shuffle

The release note for this reads:

Django test runner now supports a --shuffle option to execute tests in a random order.

I’m delighted that Django now has this option. It gives us strong protection against non-isolated tests.

When tests are not isolated, one test depends on the side effect from another. For example, take this test case:

from django.test import SimpleTestCase

from example.core.models import Book


class BookTests(SimpleTestCase):
    def test_short_title(self):
        Book.SHORT_TITLE_LIMIT = 10
        book = Book(title="A Christmas Carol")

        self.assertEqual(book.short_title, "A Chris...")

    def test_to_api_data(self):
        book = Book(title="A Song of Ice and Fire")

        self.assertEqual(
            book.to_api_data(),
            {"title": "A Song of Ice and Fire", "short_title": "A Song ..."},
        )

The tests pass when run forwards (test_short_title first), but fail in reverse. This is because test_short_title monkeypatches Book.SHORT_TITLE_LIMIT to a new value, and test_to_api_data’s expected data depends on this change.

Non isolated tests can arise all too easily in an evolving codebase. And since they’re “action at a distance”, they can be impossible to spot in code review. This leads to wasted time down the line when the isolation failure exposes itself.

Thankfully we can protect against isolation failures by running our tests in several different orders. There are two simple techniques: either reversing the order some of the time (e.g. on CI), or shuffling the order every time.

Reversing the order is effective, but it can’t detect every isolation failure.

Since shuffling the order every time eventually tries every possible order, it discovers all isolation failures, and normally in few runs. In order to allow repeating of a failing test order, we use pseudo-randomness based on a seed, like Python’s random module.

I covered test isolation in Speed Up Your Django Tests, and how to use these techniques on the two popular test frameworks. Until Django 4.0, Django only supported the weaker reverse order technique:

I’d recommend always using random test order. We can do this by default on Django 4.0+ with with a custom test runner via the TEST_RUNNER setting:

from django.test.runner import DiscoverRunner


class ExampleTestRunner(DiscoverRunner):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if self.shuffle is False:
            self.shuffle = None  # means “autogenerate a seed”

(On pytest, simply install pytest-randomly.)

Thanks to Chris Jerdonek for the contribution in Ticket #24522, and Mariusz Felisiak for reviewing.

2. --buffer with --parallel

The release note for this change reads:

Django test runner now supports a --buffer option with parallel tests.

The --buffer option is something Django inherits from unittest. When active unittest captures output during each test and only displays it if the test fails. This is something that pytest does by default.

In theory, tests should be carefully written to capture output from all the functions they call. In practice this can be hard to keep on top of as you variously add output and logging. After a while your test run can stop displaying a row of neat dots and instead spew pages of irrelevant text.

With lots of output, tests can slow down considerably: your terminal program has to store, render, and scroll all that text. On one project I saw a three times slowdown from such unnecessary output.

Baptiste Mispelon contributed Django support for --buffer in version 3.1. But due to limitations in Django’s test framework, it was not supported when using --parallel. This limited its utility when running a full test suite, as --parallel can speed up tests a bunch; (normally) more than a lack of -buffer slow them down.

From Django 4.0, thanks to some internal refactoring, we can use --buffer with --parallel.

We can thus enable --buffer by default with, again, a custom test runner class:

from django.test.runner import DiscoverRunner


class ExampleTestRunner(DiscoverRunner):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.buffer = True

This is a feature I contributed in Ticket #31370, inspired by writing Speed Up Your Django Tests. Thanks Baptiste Mispelon, Mariusz Felisiak, and Carlton Gibson for reviewing.

3. --parallel auto

Here’s the release note:

The test --parallel option now supports the value auto to run one test process for each processor core.

This is a simple tweak to this useful option.

Previously you could pass --parallel to run as many test processes as processor cores, or --parallel N for N processes. If you ran test like manage.py test --parallel example.tests, then example.tests would be interpreted as an invalid specification for N processes. It was possible to use the -- spacer, like manage.py test --parallel -- example.tests, but that is a non-obvious argparse feature.

From Django 4.0 it’s possible to specify --parallel auto so that any later arguments are correctly interpreted. This change is useful both for directly running test and for wrapper scripts.

I contributed this change in Ticket #31621. Thanks to Ahmad Hussein, Mariusz Felisiak, Tom Forbes and for reviewing. And thanks to Mariusz Felisiak, Chris Jerdonek, and Tim Graham for fixing a bug I introduced in my initial PR.

(I have an idea to extend this behaviour to support simple expressions. For example --parallel auto-2 could run 2 fewer processes than processor cores. This would prevent test runs from completely locking up the computer they’re run on.)

4. Automatic disabling of database serialization

There are two release notes for this change. The first is a bit cryptic, covering the changes to the internals:

The new serialized_aliases argument of django.test.utils.setup_databases determines which DATABASES aliases test databases should have their state serialized to allow usage of the serialized_rollback feature.

The second covers the deprecation of the database setting that’s no longer required:

SERIALIZE test setting is deprecated as it can be inferred from the TestCase.databases with the serialized_rollback option enabled.

Let’s unpack this.

Normally when a TransactionTestCase rolls back the database, it leaves all tables empty. This means any data created in your migrations will be missing during such tests.

(Such a rollback can also occur in TestCase when using a non-transactional database, notably MySQL with MyISAM.)

To fix this issue, TestCase and TransactionTestCase have the serialized_rollback flag. This makes rollback reload all database contents, after flushing the tables.

To support this rollback, Django would always serialize the entire database at the start of the test run. This takes time and memory - Django stores the data in a (potentially large) in-memory string.

Since most projects do not use serialized_rollback, this up front serialization work was usually wasted. We could disable it with the SERIALIZE database test setting:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": os.path.join(BASE_DIR, "db.sqlite3"),
        "TEST": {
            "SERIALIZE": False,  # 👈
        },
    }
}

This saves a small amount of time per test run, perhaps a few seconds on larger projects. I cover this as one of many time savers in the “Easy Wins” chapter of Speed Up Your Django Tests.

From Django 4.0, Django will only serializes databases if required. The test runner inspects all the collected tests, and only serializes databases used in test cases with serialize_rollback = True.

Because of this change, most projects will get that small speed boost automatically. Additionally the surplus setting is deprecated.

Thanks to Simon Charette for working on this in Ticket #32446, and Mariusz Felisiak for reviewing.

5. Recursive on_commit() callbacks

From this release note:

TestCase.captureOnCommitCallbacks now captures new callbacks added while executing transaction.on_commit() callbacks.

I contributed TestCase.captureOnCommitCallbacks() in Django 3.2 for testing on_commit() callbacks. I blogged about it at the time, and my backport package django-capture-on-commit-callbacks.

One rarer case was unfortunately missed: when an on_commit() callback adds a further callback. For example:

def some_view(request):
    ...
    transaction.on_commit(do_something)
    ...


def do_something():
    ...
    transaction.on_commit(another_action)
    ...


def do_something_else():
    ...

Django’s normal pathway handles such “recursive” callbacks fine, executing all the callbacks appropriately. But captureOnCommitCallbacks() failed to capture them (it had one job...).

From Django 4.0, captureOnCommitCallbacks() handles recursive callbacks, allowing us to properly test such situations. This behaviour is available on previous Django versions through my backport package django-capture-on-commit-callbacks.

Thanks to Eugene Morozov for the contribution in Ticket #33054, and Mariusz Felisiak for reviewing.

6. Test runner logging

There are two release notes for this change:

• The new logger argument to DiscoverRunner allows a Python logger to be used for logging.
• The new DiscoverRunner.log method provides a way to log messages that uses the DiscoverRunner.logger, or prints to the console if not set.

Essentially, Django’s core test runner class, DiscoverRunner, can now optionally use Python’s logging framework. This is useful for customizing its output, or for testing custom runner subclasses and making assertions on its output.

Thanks to Chris Jerdonek and Daniyal Abbasi for the contributions in Ticket #32552. Thanks to Ahmad Hussein, Carlton Gibson, Chris Jerdonek, David Smith, and Mariusz Felisiak for reviewing.

Fin

I hope these new features help your tests,

—Adam