Efficient Reloading in Django’s Runserver With Watchman

2021-01-20 Go set a watchman.

If you start the development server on a Django project, it looks something like this:

$ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (1 silenced).
January 20, 2021 - 04:25:31
Django version 3.1.5, using settings 'db_buddy.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

When you make changes to a Python file, the server automatically reloads. This is powered by a file watcher, and Django reports which one it’s using in the first line, defaulting to StatReloader. The StatReloader class is simple but reliable. It works by running a loop that checks all your files for changes every second - this is pretty inefficient, especially as your project grows!

A lesser-known but better alternative is to use Django’s support for watchman. Support was added in Django 2.2, thanks to Tom Forbes in Ticket #27685.

Watchman is an efficient file watcher open sourced by Facebook. It works by receiving file change notifications from your operating system and bundling them together. When nothing is changing, it doesn’t need to do any work - saving processing power and consequently laptop battery life. And when something changes, Django gets notified about it in milliseconds. Watchman is also smart with changes, batching them together, and integrating with Git to wait for operations like switching branches to finish.

Setup

It’s just a few steps to get Django using watchman, covered in a couple paragraphs in the runserver documentation. Here I’ll cover the exact commands it needed me to run to set it up for DB Buddy on macOS, which is a pretty vanilla Django 3.1 project.

First, I installed watchman, following the installation instructions on its site. On macOS, this meant just running brew install watchman.

Second, I installed pywatchman, the library Django uses to interface with watchman. I use pip-compile from pip-tools to manage my requirements, so first I added pywatchman to my requirements.in:

diff --git a/requirements.in b/requirements.in
index c376421..4407458 100644
--- a/requirements.in
+++ b/requirements.in
@@ -28,2 +28,3 @@ pytest-xdist
 python-dotenv
+pywatchman
 requests

I then ran pip-compile to pin the current version of pywatchman in my requirements.txt, resulting in this change:

diff --git a/requirements.txt b/requirements.txt
index 95002f9..5d9073c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -155,2 +155,4 @@ pytz==2020.5
     #   tzlocal
+pywatchman==1.4.1
+    # via -r requirements.in
 requests-mock==1.8.0

I then ran pip install -r requirements.txt to install the compiled requirements file.

Third, I set up a watchman configuration file in order to ignore my node_modules directory. The Django documentation hints that ignoring non-Python directories like this may be sensible to reduce load. My project has a relatively minimal JavaScript setup, but that still amounts to 14,303 files in the node_modules directory, all of which don’t need watching.

I followed the watchman configuration documentation and created a .watchmanconfig file next to my manage.py containing:

{
  "ignore_dirs": ["node_modules"]
}

Fourth, I restarted runserver. Django confirmed it’s using watchman by listing WatchmanReloader as the file watcher on the first line:

$ python manage.py runserver
Watching for file changes with WatchmanReloader
Performing system checks...

System check identified no issues (1 silenced).
January 20, 2021 - 07:49:15
Django version 3.1.5, using settings 'db_buddy.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Great!

Benchmark

I followed this up with a quick before/after comparison of CPU usage, whilst the server was completely idle and not serving any requests.

Using StatReloader, runserver took about ~1.6% of a CPU core.

With WatchmanReloader, this dropped to 0%, with the watchman process also showing 0%. That’s fantastic!

This project is also relatively small at the moment, with only 50 Python files and a few dependencies. Since the StatReloader’s work is proportional to the number of files, the difference will only be more stark on larger projects.

Fin

If you’re interested in learning more about the implementation of Django’s watchman integration, check out Tom Forbes’ EuroPython talk.

May your edit cycle be ever faster and more efficient,

—Adam


Want better tests? Check out my book Speed Up Your Django Tests which teaches you to write faster, more accurate tests.


Subscribe via RSS, Twitter, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: django