What happens when you run manage.py test?2020-09-05
You run your tests with
You know what happens inside your tests, since you write them.
But how does the test runner work to execute them, and put the dots, E’s, and F’s on your screen?
When you learn how Django middleware works, you unlock a huge number of use cases, such as changing cookies, setting global headers, and logging requests. Similarly, learning how your tests run will help you customize the process, for example loading tests in a different order, configuring test settings without a separate file, or blocking outgoing HTTP requests.
In this post, we’ll make a vital customization of our test run’s output - we’ll swap the “dots and letters” default style to use emojis to represent test success, failure, etc:
But before we can write that, we need to deconstruct the testing process.
Let’s investigate the output from a test run. We’ll use a project containing only this vacuous test:
When we run the tests, we get some familiar output:
To investigate what’s going on, we can ask for more detail by increasing the verbosity to maximum with
Okay great, that’s plenty! Let’s take it apart.
The first line, “Creating test database…”, is Django reporting the creation of a test database. If your project uses multiple databases, you’ll see one line per database.
I’m using SQLite in this example project, so Django has automatically set
mode=memory in the database address.
This makes database operations about ten times faster.
Other databases like PostgreSQL don’t have such modes, but there are other techniques to run them in-memory.
The second “Operations to perform” line and several following lines are the output of the
migrate command on our test databases.
This output is identical to what we’d see running
manage.py migrate on an empty database.
Here I’m using a small project with no migrations - on a typical project you’d see one line for each migration.
After that, we have the line saying “System check identified no issues”.
This output is from Django’s system check framework, which runs a number of “preflight checks” to ensure your project is well configured.
You can run it alone with
manage.py check, and it also runs automatically in most management commands.
Typically it’s the first step before a management command, but for tests it’s deferred until after the test databases are ready, since some checks use database connections.
You can write your own checks to detect configuration bugs. Because they run first, they’re sometimes a better fit than writing a test. I’d love to go into more detail, but that would be another post’s worth of content.
The following lines cover our tests. By default the test runner only prints a single character per test, but with a higher verbosity Django prints a whole line per test. Here we only have one test, “test_one”, and as it finished running, the test runner appended its status “ok” to the line.
To signify the end of the run, there’s a divider made with many dashes: “—–”. If we had any failures or errors, their stack traces would appear before this divider. This is followed by a summary of the tests that ran, their total runtime, and “OK” to indicate the test run was successful.
The final line reports the destruction of our test database.
This gives us a rough order of steps in a test run:
- Create the test databases.
- Migrate the databases.
- Run the system checks.
- Run the tests.
- Report on the test count and success/failure.
- Destroy the test databases.
Let’s track down which components inside Django are responsible for these steps.
Django and unittest
As you may be aware, Django’s test framework extends the
unittest framework from the Python standard library.
Every component responsible for the above steps is either built in to
unittest, or one of Django’s extensions.
We can represent this with a basic diagram:
We can find the components on each side by tracing through the code.
The “test” Management Command
The first place to look is the
test management command, which Django finds and executes when we run
This lives in
As management commands go, it’s quite short - under 100 lines.
handle() method is mostly concerned with handing off to a a “Test Runner”.
Simplifying it down to three key lines:
So what’s this
It’s a Django component that coordinates the test process.
It’s customizable, but the default class, and the only one in Django itself, is
Let’s look at that next!
DiscoverRunner is the main coordinator of the test process.
It handles adding extra arguments to the management command, creating and handing off to sub-components, and doing some environment setup.
It starts like this:
(Documentation, Full source.)
These class level attributes point to other classes that perform different steps in the test process. You can see most of them are unittest components.
Note that one of them is called
test_runner, so we have two distinct concepts called “test runner” - Django’s
DiscoverRunner and unittest’s
DiscoverRunner does a lot more than
TextTestRunner and has a different interface.
Perhaps Django could have used a different name for
TestCoordinator, but it’s probably too late to change that now.
The main flow in
DiscoverRunner is in its
Stripping out a bunch of details,
run_tests() looks something like this:
That’s quite a few steps! Many of the called methods correspond with steps on our above list:
setup_databases()creates of the test databases. This only creates the databases necessary for the selected tests, as filtered by
get_databases(), so if you run only database-free
SimpleTestCases, Django doesn’t create any databases. Inside it both creates the databases and runs the
run_checks(), well, runs the checks.
run_suite()runs the test suite, including all the output.
teardown_databases()destroys the test databases.
A couple of other methods are things we may have expected:
teardown_test_environment()set and unset some settings, such as the local email backend.
suite_result()returns the number of failures to the
All these methods are useful to investigate for customizing those parts of the test process.
But they’re all part of Django itself.
The other methods hand off to components in
Let’s investigate those in turn.
build_suite() is concerned with finding the tests to run, and putting them into a “suite” object.
It’s again a long method, but simplified, it looks something like this:
This method uses three of the four classes that we saw
DiscoverRunner refers to:
test_suite- a unittest component that acts as a container for tests to run.
parallel_test_suite- a wrapper around a test suite used with Django’s parallel testing feature.
test_loader- a unittest component that can find test modules on disk, and load them into a suite.
DiscoverRunner method to look into is
We don’t need to simplify this one - its entire implementation looks like this:
Its only concern is constructing a test runner and telling it to run the constructed test suite.
This the final one of the
unittest components referred to by a class attribute.
unittest.TextTestRunner, which is the default runner for outputting results as text, as opposed to, for example, an XML file to communicate results to your CI system.
Let’s finish our investigation by looking inside that class.
This component inside unittest takes a test case or suite, and executes it. It starts:
DiscoverRunner, it uses a class-level attribute to refer to another class.
TextTestResult class is the thing that actually writes the text-based output.
DiscoverRunner’s class references, we can override
resultclass by passing an alternative to
We’re now ready to customize the test process. But first, let’s review our investigation.
We can now expand our map to show the classes we’ve found:
There’s certainly more detail we could add, such as the contents of several important methods in
But what we have is by far enough to implement many useful customizations.
How to Customize
Django offers us two ways to customize the test running process:
- Override the test command with a custom subclass.
- Override the
DiscoverRunnerclass by pointing the
TEST_RUNNERsetting at a custom subclass.
Because the test command is so simple, most of the time we’ll customize by overriding
DiscoverRunner refers to the unittest components via class-level attributes, we can replace them by redefining the attributes in our custom subclass.
Super Fast Test Runner
For a basic example, imagine we want to skip all our tests and report success every time.
We can do this by creating a
DiscoverRunner subclass with a new
run_tests() method that doesn’t call its
Then we use it in our settings file like so:
When we run
manage.py test now, it completes in record time!
Great, that is very useful.
Let’s finally get on to our much more practical emoji example.
From our investigation we found that the
TextTestResult is responsible for performing the output.
We can replace it in
DiscoverRunner, by having it pass a value for
Django already has some options to swap
resultclass, for example the
--debug-sql option which prints executed queries for failing tests.
TextTestRunner with arguments from the
This in turn calls
get_resultclass(), which returns a different class if one of two test command options (
--pdb) have been used:
If neither option is set, this method implicitly returns
TextTestResult to use the default
We can detect this
None in our custom subclass and replace it with our own
EmojiTestResult class extends
TextTestResult and replaces the default “dots” output with emoji.
It ends up being quite long since it has one method for each type of test result:
After pointing the
TEST_RUNNER setting at
EmojiTestRunner, we can run tests and see our emoji:
After our spelunking, we’ve seen the
unittest design is relatively straightforward.
We can swap classes for subclasses to change any behaviour in the test process.
This works for some project-specific customizations, but it’s not very easy to combine others’ customizations.
This is because the design uses inheritance and not composition.
We have to use multiple inheritance across the web of classes to combine customizations, and whether this works depends very much on the implementation of the customizations.
Because of this not really a plugin ecosystem for
I know of only two libraries providing custom
- unittest-xml-reporting - provides XML output for your CI system to read.
- django-slow-tests - provides measurement of test execution time, to find your slowest tests.
I haven’t tried, but combining them may not work, since they both override parts of the output process.
In contrast, pytest has a flourishing ecosystem with over 700 plugins. This is because its design uses composition with hooks, which act similar to Django signal. Plugins register for only the hooks they need, and pytest calls every registered hook function at the corresponding point of the test processes. Many of pytest’s built-in features are even implemented as plugins.
If you’re interested in heavier customization of your test process, do check out pytest.
Thank you for joining me on this tour. I hope you’ve learned something about how Django runs your tests, and can make your own customizations,
Working on a Django project? Check out my book Speed Up Your Django Tests which covers loads of best practices so you can write faster, more accurate tests.
One summary email a week, no spam, I pinky promise.
- Make Django Tests Always Rebuild the Database if It Exists
- How to Unit Test a Django Form
- Introducing time-machine, a New Python Library for Mocking the Current Time
© 2020 All rights reserved.