Book-Driven Development from “Boost Your Django DX”
On Monday I released my new book “Boost Your Django DX”. It covers many tools and practices that are useful for developing Django projects.
Whilst writing I often try to describe something and find it hard to explain, or suboptimal in some way. Since I want to make things as easy as possible for readers, this process leads me to make open source improvements, or new packages. I have taken to calling this process Book-Driven Development.
Here’s a list of the more interesting packages and contributions I made whilst writing this book.
Chapter 5 of the book covers ways to enhance Django’s development server,
runserver. Whilst writing it, I took some time to research tools for automatically reloading the web browser files change. I knew of some tools out there, I had just never seen any that were easy to set up. Most relied on running an extra Node.js process and proxy server, which felt like way too much for my tastes.
After a little bit of digging, I decided to try my hand at writing my own browser reloader. My proof-of-concept worked so well that I spent the next day turning it into a package. And so, I created django-browser-reload.
You can read a bit more about django-browser-reload in my introductory post.
After releasing the package I found that frequent Django contributor Keryn Knight has a similar package called django-livereloadish. It’s a bit more experimental, but there are some ideas I’d like to merge into django-browser-reload, such as saving and restoring page state.
Rich is a wildly popular package for pretty terminal output. In the book I describe using it with Django, both for logging and within management commands.
The logging use case is easy enough, as Rich seamlessly integrates with the
logging module. Django configures
logging with its
Management commands are a little trickier though. In command code, it’s best to use Django’s
stderr wrappers, rather than calling
print() directly. These wrappers improve testability: Django’s
call_command() lets you use them to to capture output for assertions.
I started writing a section on using Rich’s
Console class with the output wrappers, but found that covering all the cases is a bit involved. It was too much code to tell readers to copy and paste, so I put the code into a new package, django-rich. The package provides a command class that sets up the
Console for you:
from django_rich.management import RichCommand class Command(RichCommand): def handle(self, *args, **options): self.console.print("[bold red]Alert![/bold red]")
The package may grow to cover more ways of using Rich with Django. I wrote a couple of ideas in its issues list.
Before I started properly working on the book, I was thinking about tools to improve DX. I really enjoy using pyupgrade, which rewrites code using old Python syntax to new. It’s handy when run as a pre-commit hook, as it can then update new code added that accidentally uses old features.
django-codemod is a similar tool for upgrading code using old Django features to new ones. I’d used django-codemod and it works great. But it is unfortunately slow—it could take two minutes to process a medium-sized project, with all CPU cores a-blazing. It is therefore too slow to practically run as a pre-commit hook.
I tried experimenting with rebuilding django-codemod with the same underlying approach as pyupgrade. This worked well, so I turned the experiment into django-upgrade. I since added more fixers, and released the tool, and it’s been fairly popular.
You can read a bit more about django-upgrade in my introductory post. The book covers both pyupgrade and django-upgrade.
I want to keep my book as small as possible. The largest part of its file size, in any format, is the images. Most of these are PNG screenshots demonstrating.
PNG’s can be easily optimized, especially my screenshots (macOS seems to prioritize quick saving over a smaller filesize). Often the savings are more than 50%!
I wanted an easy way to run oxipng under pre-commit, and so I created repository pre-commit-oxipng. This repository contains the appropriate pre-commit configuration, and it automatically syncs oxipng releases thanks to pre-commit-mirror-maker. This way there’s an easy public repository for shared configuration, but the oxipng repository doesn’t need to be concerned with pre-commit. (I still opened an oxipng issue to ask if they’d like to host the config.)
django-debug-toolbar is a fantastic tool for development. It’s very popular: according to the 2021 survey, 26% of Django developers put it in their “top five” favourite packages.
My book describes setting up the toolbar and using it for several common tasks. Whilst writing this section, I found myself looking at the package’s installation instructions and source code, for the first time in a while. I made several PR’s based on this exploration:
I found several ways to improve the writing to make it clearer and briefer.
For an N+1 queries page in my test app, these two changes add up to ~30% reduction in total run time.
I know some developers can’t keep the toolbar always-on because of its overhead, so hopefully this optimization helps in those situations. From looking at profiles and code, there is definitely more room for optimization.
PR #1535 reduced the ways of running Flake8 and Black to only pre-commit.
Previously the repository also had configuration to run these tools with make and tox. I think this was a case of developers adding alternatives over the years but never pruning them. These alternatives had the possibility of installing different versions, which could cause code inconsistencies.
pre-commit is a great tool for running code quality tools, and it has wide adoption in the Python community. There are three chapters in my book covering setting up pre-commit, using it to run various tools, and even writing custom tools.
PR #1559 moved the “settings reset” function to the package’s settings module.
Thisi function receives Django’s
setting_changedsignal and uses it to clear caches based on setting values:
from django.dispatch import receiver from django.test.signals import setting_changed ... @receiver(setting_changed) def update_toolbar_config(*, setting, **kwargs): """ Refresh configuration when overriding settings. """ if setting == "DEBUG_TOOLBAR_CONFIG": get_config.cache_clear() elif setting == "DEBUG_TOOLBAR_PANELS": ...
Settings may change during tests when using e.g. the
Previously this function lived only in the django-debug-toolbar test suite. That is fine for the toolbar’s own test suite, but it meant it wouldn’t apply in test suites of projects using the toolbar.
By placing the reset function within the main package, the settings are reset whenever they’re changed, not just within the package tests. This follows the same pattern that Django uses.
PR #1560 avoided injecting the toolbar HTML when the response
Content-Encodingheader is set.
The toolbar tries to inject its HTML into all
text/htmlresponses. Previously it avoided doing so if the response had the header
gzipisn’t the only possible encoding though, there’s also e.g.
The PR changed the logic to avoid injecting the HTML whenver the response has any encoding, because it cannot decompress the contents. (In most setups, responses will be compressed after the toolbar HTML is injected—the settings even guide you to do so.)
There are still more ways to improve django-debug-toolbar, so some more PR’s may be forthcoming soon!
I write my book with Sphinx, the primary documentation tool in the Python ecosystem. It’s used to document Python, Django, and most projects on Read the Docs.
To check my spelling, I use the sphinxcontrib-spelling extension. I found out about this package because it’s used for the Django docs.
To run the spell check automatically, I run it with pre-commit. This prevents me from even committing misspelled words. Unfortunately I found that as I wrote more, this process took longer and longer.
I profiled the spell check process, and discovered that most of the time was spent in a class called
ImportableModuleFilter. This class checks if each word names an importable Python module, and if so, prevents the word being counted as a spelling mistake. After reading the source I found some optimizations that I made in PR #129.
The optimizations helped, but only reduced the runtime by about 10%. So I decided to disable several of the spell check filters, with these settings:
spelling_lang = "en_GB" tokenizer_lang = "en_GB" spelling_ignore_acronyms = False spelling_ignore_contributor_names = False spelling_ignore_importable_modules = False spelling_ignore_wiki_words = False
This change took more than 50% off the runtime. The only downside is that when mentioning importable modules for the first time, I have to add them to by spelling “allow list” file. I find that pretty acceptable.
Learn how to make your tests run quickly in my book Speed Up Your Django Tests.
One summary email a week, no spam, I pinky promise.
Tags: django, python