How to Use Django's Parallel Testing on macOS With Python 3.8+

2020-07-21 Fork or Spoon?

This post is an adapted extract from my book Speed Up Your Django Tests, available now.

Django’s test command takes the --parallel flag to activate parallel testing. Unfortunately, this has never worked on Windows, and with Python 3.8 it has stopped working on macOS.

Django uses the “fork” method in multiprocessing to start the new test processes. Windows does not support “fork”, and whilst macOS does, Python 3.8 changed its default to “spawn”. If the start method is not “fork” when you run tests with the --parallel flag, Django silently switches back to serial testing.

The “spawn” method works quite differently to “fork”, requiring a lot of initialization in the new test processes, so the current test framework is incompatible. This is a known issue, Ticket #31169, and it is being worked on by Ahmad A. Hussein as part of Django’s Google Summer of Code 2020. Hopefully we will merge the fix in Django 3.2.

For the time being, there’s a workaround for macOS. Alternatively, you can switch to pytest, since its parallelization plugin, pytest-xdist, works on all platforms.

On macOS on Python 3.8+, you can work around the lack of “spawn” support by reverting the multiprocessing start method to “fork”. This is only a temporary workaround, and has a small risk of introducing system library crashes to your test suite. Python 3.8 changed the macOS default because many system libraries crash when used after forking (Python Issue #33725).

The first step in the workaround is to disable the macOS system protection against forking. Do this by setting this environment variable in the terminal you run tests:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

You can do this in your shell initialization file so it always applies - see this Stack Overflow answer.

The second step is to add a call to multiprocessing.set_start_method() at the top of main() in your manage.py file. This is best accompanied with a check that OBJC_DISABLE_INITIALIZE_FORK_SAFETY has indeed been set:

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import multiprocessing
import os
import sys


def main():
    try:
        command = sys.argv[1]
    except IndexError:
        command = "help"

    if command == "test" and sys.platform == "darwin":
        # Workaround for https://code.djangoproject.com/ticket/31169
        if os.environ.get("OBJC_DISABLE_INITIALIZE_FORK_SAFETY", "") != "YES":
            print(
                (
                    "Set OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in your"
                    + " environment to work around use of forking in Django's"
                    + " test runner."
                ),
                file=sys.stderr,
            )
            sys.exit(1)
        multiprocessing.set_start_method("fork")
    ...

The check will stop manage.py test from starting on macOS if OBJC_DISABLE_INITIALIZE_FORK_SAFETY has not been set.

Fin

May your tests run fast in parallel,

—Adam


Working on a Django project? Check out my book Speed Up Your Django Tests which covers loads of best practices so you can write faster, more accurate tests.


Subscribe via RSS, Twitter, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: django