The Fast Way to Test Django transaction.on_commit() Callbacks

Django’s transaction.on_commit() hook is useful for running tasks that rely on changes in the current database transaction. The database connection enqueues callback functions passed to on_commit, and executes the callbacks after the current transaction commits. If the transaction is rolled back, the callbacks are discarded. This means they act if-and-when the final version of the data is visible to other database connections.

It’s a best practice to use on_commit for things like sending external emails or enqueueing Celery tasks. (See my previous post Common Issues Using Celery (And Other Task Queues).)

Unfortunately, testing callbacks passed to on_commit() is not the smoothest. The Django documentation explains the problem:

Django’s TestCase class wraps each test in a transaction and rolls back that transaction after each test, in order to provide test isolation. This means that no transaction is ever actually committed, thus your on_commit() callbacks will never be run. If you need to test the results of an on_commit() callback, use a TransactionTestCase instead.

TransactionTestCase is correct and works for such tests, but it’s much slower than TestCase. Its rollback behaviour flushes every table after every test, which takes time proportional to the number of models in your project. So, as your project grows, all your tests using TransactionTestCase get slower.

I cover this in my book Speed Up Your Django Tests in the section “TestCase Transaction Blockers”. on_commit() is one thing that can force you to use TransactionTestCase, “blocking” you from the speed advantages of TestCase. Thankfully there’s a way to test them using TestCase, with a little help from a targeted mock.

Django Ticket #30457 proposes adding a function for running on_commit callbacks inside TestCase. I’ve used a snippet similar to those posted on the ticket in several client projects, so I figured it was time to pick up the ticket and add it to Django core. I’ve thus made PR #12944 with TestCase.captureOnCommitCallbacks().

My PR is awaiting review and (hopefully) a merge, and it targets Django 3.2 which will be released nearly a year from now in April 2021. I’ve thus released it in a separate package django-capture-on-commit-callbacks, available now for Django 2.0+.

Update (2020-07-15): My PR has been merged, so my package can now be considered a backport of the official feature.

After you’ve installed it and added it to your project’s custom TestCase class, you can use it like so:

from django.core import mail
from example.test import TestCase


class ContactTests(TestCase):
    def test_post(self):
        with self.captureOnCommitCallbacks(execute=True) as callbacks:
            response = self.client.post(
                "/contact/",
                {"message": "I like your site"},
            )

        self.assertEqual(response.status_code, 200)
        self.assertEqual(len(callbacks), 1)
        self.assertEqual(len(mail.outbox), 1)
        self.assertEqual(mail.outbox[0].subject, "Contact Form")
        self.assertEqual(mail.outbox[0].body, "I like your site")

These tests POST to a view at /contact that uses an on_commit() callback to send an email. Passing execute=True to captureOnCommitCallbacks() causes it to execute the captured callbacks as its context exits. The assertions are then able to check the HTTP response, the number of callbacks enqueued, and the sent email.

For more information see django-capture-on-commit-callbacks on PyPI.

If your Django project’s long test runs bore you, I wrote a book that can help.

One summary email a week, no spam, I pinky promise.

Related posts:

• Use partial() With Django’s transaction.on_commit() to Avoid Late-Binding Bugs
• Common Issues Using Celery (And Other Task Queues)
• Django’s Test Case Classes and a Three Times Speed-Up

Tags: django, python