Django: Introducing inline-snapshot-django

Snapshot testing and inline-snapshot

inline-snapshot-django builds on top of the excellent inline-snapshot, a nifty snapshot testing library.

Snapshot testing is a type of testing that compares a result against a previously captured “snapshot” value. It can enable you to write tests quickly and assert on many details that you'd otherwise miss.

Snapshot testing is quite popular in other languages, such as JavaScript, but it has been underused in Python. This may be due to a lack of good tools to help automate snapshot management. inline-snapshot provides nice workflows tied into pytest, so I hope it will popularize the technique among Python and Django developers.

The inline-snapshot developer, Frank Hoffmann, is working intensely on it, and he provides some extra tools and features for his GitHub sponsors.

inline-snapshot-django

I created inline-snapshot-django to provide an advanced yet easier-to-use alternative to Django’s assertNumQueries(). While assertNumQueries() is handy, it is a rather blunt tool that leaves you in the dark when debugging failures. For example, say you encountered this test:

from django.test import TestCase


class IndexTests(TestCase):
    def test_success(self):
        with self.assertNumQueries(1):
            response = self.client.get("/")

        assert response.status_code == 200

If it starts failing with an incorrect query count, you’ll see an error like:

$ pytest
========================= test session starts =========================
...
example/tests.py F                                              [100%]

============================== FAILURES ===============================
_______________________ IndexTests.test_success _______________________

self = <example.tests.IndexTests testMethod=test_success>

    def test_success(self):
>       with self.assertNumQueries(1):
             ^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError: 2 != 1 : 2 queries executed, 1 expected
E       Captured queries were:
E       1. SELECT COUNT(*) AS "__count" FROM "example_character"
E       2. SELECT "example_character"."id", "example_character"."name", "example_character"."strength", "example_character"."charisma", "example_character"."klass_id" FROM "example_character" LIMIT 10

The message makes it clear that the query count is incorrect, but it doesn’t help you narrow down what changed. You might glean the change from the captured queries, but without a record of what the queries were before, it’s hard to know what to look for. This problem becomes particularly challenging in larger projects with many queries.

One client project that I have been working on tried to alleviate this problem by adding a docstring under the assertNumQueries() call, recording the expected query fingerprints:

from django.test import TestCase


class IndexTests(TestCase):
    def test_success(self):
        with self.assertNumQueries(1):
            """
            1. SELECT ... FROM example_character
            """
            response = self.client.get("/")

        assert response.status_code == 200

This was a great idea, but failures still required manual comparison and updating of the query fingerprints. Managing and checking these fingerprints required a considerable effort, particularly when I worked on a Django 5.2 upgrade that added and removed some SQL queries in the admin interface. That work inspired inline-snapshot-django to automate such fingerprint comments, using inline-snapshot.

Here’s the above test rewritten using inline-snapshot-django:

from django.test import TestCase
from inline_snapshot import snapshot
from inline_snapshot_django import snapshot_queries


class IndexTests(TestCase):
    def test_success(self):
        with snapshot_queries() as snap:
            response = self.client.get("/")

        assert response.status_code == 200
        assert snap == snapshot(
            [
                "SELECT ... FROM example_character LIMIT ...",
            ]
        )

The test now uses the snapshot_queries() context manager to capture the fingerprints of the executed SQL queries. The fingerprints are then compared with a snapshot() call in the final assertion.

inline-snapshot manages values within snapshot() calls. It wraps comparisons, and if they fail, it can automatically update the call inside the test file with the new value.

For example, if we repeat the previous failure, inline-snapshot will prompt us to update the snapshot:

$ pytest
...

example/tests.py .E                                             [100%]

═══════════════════════════ inline-snapshot ═══════════════════════════
──────────────────────────── Fix snapshots ────────────────────────────
╭───────────────────────── example/tests.py ──────────────────────────╮
│ @@ -11,6 +11,7 @@                                                   │
│                                                                     │
│          assert response.status_code == 200                         │
│          assert snap == snapshot(                                   │
│              [                                                      │
│ +                "SELECT ... FROM example_character",               │
│                  "SELECT ... FROM example_character LIMIT ...",     │
│              ]                                                      │
│          )                                                          │
╰─────────────────────────────────────────────────────────────────────╯
Do you want to fix these snapshots? [y/n] (n):

As the message implies, answering “y” updates the test file to insert the new query fingerprint:

@@ -11,6 +11,7 @@
         assert response.status_code == 200
         assert snap == snapshot(
             [
+                "SELECT ... FROM example_character",
                 "SELECT ... FROM example_character LIMIT ...",
             ]
         )

inline-snapshot only does this prompting when running in an interactive terminal. On non-interactive runs, such as in CI, it will simply fail the test.

inline-snapshot also provides other modes, such as --inline-snapshot=fix, to automatically fix all snapshots without prompting. I covered the workflows for such modes with inline-snapshot-django over in its usage documentation.

History and fun with Rust

inline-snapshot-django is a new package, but it’s inspired by my previous package django-perf-rec (Django performance recorder). I created django-perf-rec nearly 9 years ago, in 2016, to perform similar SQL query snapshot testing. However, it works differently by storing the snapshots in a separate YAML file.

For example, you might write the previous test with django-perf-rec like:

import django_perf_rec
from django.test import TestCase


class IndexTests(TestCase):
    def test_success(self):
        with django_perf_rec.record():
            response = self.client.get("/")

        assert response.status_code == 200

The record() context manager captures SQL query fingerprints and stores them in an entry in a YAML file, copying the name from the test file, such as tests.perf.yml. Its contents would look like:

IndexTests.test_success:
- db: SELECT COUNT(*) AS "__count" FROM "example_character"
- db: 'SELECT ... FROM "example_character" LIMIT #'

While it can be nice to get long query fingerprint lists out of the test file, the old package has many disadvantages:

• Separate snapshots are harder to review, as they don’t make it super clear what changed, where.
• Renaming test files, classes, or methods requires manually updating the YAML file.
• YAML syntax is not always obvious and can be hard to format nicely.
• The SQL fingerprinting process was relatively slow and inflexible, being built in Python on top of sqlparse.

Starting afresh in inline-snapshot-django allowed me to address these issues. Building on top of inline-snapshot solves the first three issues, as it stores the snapshots inline in the test file. I tackled the final issue of poor SQL fingerprinting by building a new SQL fingerprinting library in Rust, called sql-fingerprint.

sql-fingerprint is my second open source Rust project, after Djade, my Django template formatter. It provides a fast way to fingerprint SQL queries, based on the excellent sqlparser crate, a production-grade SQL parser used by many Rust-based databases and tools.

I made sql-fingeprint as a Rust crate, and intended to publish it to PyPI as a Python package with the same name. However, at the last second, PyPI rejected the name sql-fingerprint because it was too similar to an existing package, sqlfingerprint (no hyphen). Therefore, the Python wrapper of sql-fingerprint is called sql-impressao, based on the Portuguese word for “fingerprint” (impressão digital).

Using Rust has worked well for sql-fingerprint/sql-impressao, as it is much faster to process strings in Rust than in Python. While the previous SQL fingerprinting in django-perf-rec would be rather visible in test profiles, taking perhaps 1-5% of the total test time. sql-impressao rounds to 0% in the test profiles I’ve done.

Response snapshots would be cool too

While inline-snapshot-django currently only supports SQL query fingerprints, I think it can be extended to support snapshot testing of other Django features. For example, it might be possible to snapshot response details from the test client, allowing instant tests like:

from django.test import TestCase
from inline_snapshot import snapshot
from inline_snapshot_django import details, ResponseDetails


class IndexTests(TestCase):
    def test_success(self):
        response = self.client.get("/")

        assert details(response) == snapshot(
            ResponseDetails(
                status_code=200,
                content_type="text/html",
            )
        )

This idea is tracked in Issue #11 if you’d like to weigh in on it.

Fin

Please give inline-snapshot-django a try today in your Django projects. The documentation awaits your perusal, and I’d love to hear your feedback.

Happy snapshot testing,

—Adam