How to Limit Test Time in Django’s Test Framework2021-01-25
I recently optimized a client project’s test suite, and in the process found a test whose runtime had crept up ever since it had been written. The problematic test exercised an import process from a fixed past date until the current day. The test’s runtime therefore grew every day, until it reached over a minute.
The solution for that test was to add a date mock using time-machine, so the test created a fixed amount of data. But we also wanted to prevent such a problem occurring again.
We came up with the idea of implementing a test time limit that would automatically fail any long running tests. This way, if any appeared again, they would be detected early.
The project uses Django’s test framework, which is based on unittest.
Based on my spelunking of the unittest’s internals, I figured the best place to add such a limit would be in the
The project already had its own customized subclasses of Django’s
TestCase classes, so I could add the logic there.
Cutting out other details, here is the implementation I came up with:
The mixin wraps the unittest internal
_callTestMethod(), which is called for each test, with the time check.
Regardless of whether a test passes or fails, if it takes longer than the limit, a
SlowTestException is raised, which fails the test with an error.
I checked what this looks like with a dummy slow test that runs
One thing this started to reveal was that when running tests in parallel mode, there would often be a handful of slow tests. But when run individually or in non-parallel mode, those tests didn’t come near the time limit.
This seemed to be cause by resource starvation on the CPU or inside PostgreSQL. The solution was to reduce the default number of parallel test processes from 8 to 4.
Tests stopped hitting the time limit and the time for the whole test run was reduced a bit, which was nice to see.
Setting a limit like this is simple and will prevent slow tests being added to the project. But it still waits for the whole, slow test to complete before failing it. This would be problematic for tests that could sometimes take very long to complete.
Whilst it’s best to avoid creating such tests in the first place, if you have them you might want a fuller timeout solution that stops test as soon as they hit the limit.
This is possible using a thread or, on Unix, requesting an interrupt from the
The pytest-timeout implements both these methods and some code can probably be copied from there to unittest-based projects.
May your test suite ever expand its coverage and reduce its duration,
Want better tests? Check out my book Speed Up Your Django Tests which teaches you to write faster, more accurate tests.
One summary email a week, no spam, I pinky promise.
Tags: django, python
© 2021 All rights reserved.