Django’s Test Case Classes and a Three Times Speed-Up
This is a story about how I sped up a client’s Django test suite to be three times faster, through swapping the test case class in use.
Speeding up test runs is rarely a bad thing. Even small teams can repeat their test run hundreds of times per week, so time saved there is time won back. This keeps developers fast, productive, and happy.
A quick refresher on how the Django’s three basic test case classes affect the database:
SimpleTestCaseis the simplest one.
- It provides the basic features of
unittest.TestCaseplus some Django extras. It blocks database access by default, because it doesn’t do anything to isolate changes you would make there. You should use it for testing components that don’t need the database.
SimpleTestCaseto allow database modifications.
- It resets the database at the end by removing all rows from all database tables. This is slow, but reliable. You should use it when you need the database, but you can’t use
TestCaseis the class you should normally use.
- It extends
TransactionTestCaseand replaces its database reset process with one that uses transactions. This is much faster as the database only undoes the changes made during the test, rather than checking each table individually. It means your tests can’t commit, but in practice most tests don’t need that.
The distinction between
TestCase can be confusing. Here’s my attempt to summarize it in one sentence:
TransactionTestCasethat allows your code to use transactions, while
TestCaseuses transactions itself.
Recently I was helping my client ev.energy improve their Django project. A full test run took about six minutes on my laptop when using the test command’s
--parallel option . This isn’t particularly long - I’ve worked on projects where it took up to 30 minutes! But it did give me a little time during runs to look for easy speed-ups.
Their project uses a custom test case class for all their tests, to add extra helper methods. It originally extended
TransactionTestCase, with its slower but more complete database reset procedure. I wondered why this had been done.
I searched the Git history for the first use of
git log -S TransactionTestCase (a very useful Git option!). I found a developer had first used it in tests for their custom background task class called
Task closed the database connection at the end of its process with
`connection.close() <https://www.python.org/dev/peps/pep-0249/#Connection.close>`__. This helped isolate the tasks. Since they’re run in a long running background process, using a fresh database connection for each task helped prevent a failure in one from affecting the others.
Unfortunately the call to
connection.close() prevented use of
TestCase when testing
Task classes. Closing the database connection also ends any transactions. So when
TestCase ran its teardown process, it errored when trying to roll back the transactions it started in its setup process.
Because of this, the developers used
TransactionTestCase for their custom test case class. And they stuck with it as the project grew.
This was all fair, and the speed difference would not have been noticeable when there were fewer tests. Fixing it then allowed them to focus on feature development.
But as with test time things like this, the seconds added up over time. Much like the metaphorical frog in a slowly boiling pot of water.
Once I’d discovered this piece of history, I guessed most of the tests that didn’t run
Task classes would work with
TestCase. I swapped the base of the custom test class to
TestCase, reran, and only the
Task tests failed!
After changing only broken test classes back to
TransactionTestCase, I reran the suite and everything passed. The run time went down from 375 seconds to 120 seconds. A three times speed-up!
I hope this post helps you find the right test case class in your Django project. If you want help with this, email me - I’m happy to answer any questions, and am available for contracts. See my front page for details.
Thanks for reading,
Improve your Django develompent experience with my new book.
One summary email a week, no spam, I pinky promise.
- Introducing django-perf-rec, our Django performance testing tool
- Getting a Django Application to 100% Test Coverage