Django: introducing django-crawl

I recently migrated one of my client projects from the legacy django-csp package to Django 6.0’s built-in Content Security Policy (CSP) support (release note). This security header is a powerful tool for preventing unwanted content from being loaded on your site, so configuration correctness is paramount. The migration was fairly straightforward, but a few pages had complicated overrides, so I wanted to be sure that no CSP headers had been changed by my swapping of CSP implementations.

I had the idea to verify no page had changed its content-security-policy header by crawling the site with Django’s test client, outputting URL and header contents during the process. By diffing the output from crawls before and after the migration, I could check for changes and track down which pages had been affected.

The core loop of that script looked something like this:

from collections import deque
from django.test import Client


client = Client()
client.force_login(superuser)

queue: deque[str] = deque(["/", "/admin/"])
...

while queue:
    url = queue.popleft()
    ...
    response = client.get(url, follow=False)
    ...
    print(f"{url}\t{response.headers.get('content-security-policy')}")
    ...
    for anchor in BeautifulSoup(response.content, "html.parser").find_all(
        "a", href=True
    ):
        # Enqueue these found links
        ...

This simple crawl of the site ended up flushing out seven non-CSP bugs, despite the project having 100% test coverage and a full suite of integration tests. Those bugs were due to incorrect link generation, admin features not being disabled, and regular old broken code.

I was pretty impressed with the power of this technique for finding broken stuff! Given this experience, I wanted to expand the script into a reusable tool, which I have now done with django-crawl.

To use django-crawl, install it, add it to INSTALLED_APPS, and you can run the crawl management command:

$ ./manage.py crawl -v 2
🐛 Crawling up to 1000 URLs, logged in as Ad Min
/
/about/
/blog/
/contact/
/dev/
/blog/2026/
/blog/2025/
/blog/2026/07/22/introducing-django-crawl/
...
🦋 Crawled 1000 URLs, encountered 0 errors, stopped due to reaching max URL limit of 1000.

The command reports any errors it encounters, from broken links to exceptions. The output uses Rich for pretty formatting and a live spinner while it runs. Adding -v 2 prints the URLs as they are crawled.

The crawler discovers links in various forms in HTML responses (<a href>, <link href>, <script src>, <img src>, etc.), sitemaps, and feeds. If you have a sitemap, you can start your crawl with a simple:

$ ./manage.py crawl /sitemap.xml

The HTML parsing is done with a custom Rust extension built with html5ever, the HTML parser from the Servo project, so it’s very fast. And with no overhead from real HTTP requests or inter-process communication, the crawl is limited only by how fast your application code can run.

django-crawl also provides a Python API that you can use to make a mega-test that crawls your whole site with example data, raising an ExceptionGroup if any errors are encountered:

from django.contrib.auth.models import User
from django.test import TestCase

import django_crawl


class CrawlTests(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.admin = User.objects.create_superuser(username="admin")

    def test_crawl(self):
        client = django_crawl.CrawlClient()
        client.force_login(self.admin)
        django_crawl.crawl("/", "/admin/", client=client)

I am not sure if this is a good fit for most projects, since it will leave you with one long test that exercises many views. But it might be good for building a “safety net” on untested projects, something I know Jeff Triplett likes to do (ref Django Chat #24).

😸😸😸 Check out my new book on using GitHub effectively, Boost Your GitHub DX! 😸😸😸

One summary email a week, no spam, I pinky promise.

Related posts:

• Django: introducing django-integrity-policy
• Django: Introducing django-http-compression

Tags: django