Django’s Test Case Classes and a Three Times Speed-Up

Django’s Test Case Classes

A quick refresher on how the Django’s three basic test case classes affect the database:

• SimpleTestCase is the simplest one.

  It provides the basic features of unittest.TestCase plus some Django extras. It blocks database access by default, because it doesn’t do anything to isolate changes you would make there. You should use it for testing components that don’t need the database.

• TransactionTestCase extends SimpleTestCase to allow database modifications.

  It resets the database at the end by removing all rows from all database tables. This is slow, but reliable. You should use it when you need the database, but you can’t use TestCase…

• TestCase is the class you should normally use.

  It extends TransactionTestCase and replaces its database reset process with one that uses transactions. This is much faster as the database only undoes the changes made during the test, rather than checking each table individually. It means your tests can’t commit, but in practice most tests don’t need that.

The distinction between TransactionTestCase and TestCase can be confusing. Here’s my attempt to summarize it in one sentence:

TransactionTestCase that allows your code to use transactions, while TestCase uses transactions itself.

The Speed-Up Story

Recently I was helping my client ev.energy improve their Django project. A full test run took about six minutes on my laptop when using the test command’s --parallel option . This isn’t particularly long - I’ve worked on projects where it took up to 30 minutes! But it did give me a little time during runs to look for easy speed-ups.

Their project uses a custom test case class for all their tests, to add extra helper methods. It originally extended TransactionTestCase, with its slower but more complete database reset procedure. I wondered why this had been done.

I searched the Git history for the first use of TransactionTestCase with git log -S TransactionTestCase (a very useful Git option!). I found a developer had first used it in tests for their custom background task class called Task.

Task closed the database connection at the end of its process with connection.close(). This helped isolate the tasks. Since they’re run in a long running background process, using a fresh database connection for each task helped prevent a failure in one from affecting the others.

Unfortunately the call to connection.close() prevented use of TestCase when testing Task classes. Closing the database connection also ends any transactions. So when TestCase ran its teardown process, it errored when trying to roll back the transactions it started in its setup process.

Because of this, the developers used TransactionTestCase for their custom test case class. And they stuck with it as the project grew.

This was all fair, and the speed difference would not have been noticeable when there were fewer tests. Fixing it then allowed them to focus on feature development.

But as with test time things like this, the seconds added up over time. Much like the metaphorical frog in a slowly boiling pot of water.

Once I’d discovered this piece of history, I guessed most of the tests that didn’t run Task classes would work with TestCase. I swapped the base of the custom test class to TestCase, reran, and only the Task tests failed!

After changing only broken test classes back to TransactionTestCase, I reran the suite and everything passed. The run time went down from 375 seconds to 120 seconds. A three times speed-up!

Fin

I hope this post helps you find the right test case class in your Django project. If you want help with this, email me - I’m happy to answer any questions, and am available for contracts. See my front page for details.

Update (2019-07-05): Fellow Django core contributor Luke Plant shared a similar story on Twitter. He saw a test suite go from 20 minutes down to 7 - about three times faster again!

Thanks for reading,

—Adam