Efficient Reloading in Django’s Runserver With Watchman

Update (2022-04-06): pywatchman 1.4.1 does not work with Python 3.10. There is a a fix, but unfortunately Facebook have not released it since the issue was reported 2021-11-02. Check the upstream issue before proceeding on Python 3.10+.
If you start the development server on a Django project, it looks something like this:
$ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (1 silenced).
January 20, 2021 - 04:25:31
Django version 3.1.5, using settings 'db_buddy.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
When you make changes to a Python file, the server automatically reloads. This is powered by a file watcher, and Django reports which one it’s using in the first line, defaulting to StatReloader
. The StatReloader
class is simple but reliable. It works by running a loop that checks all your files for changes every second - this is pretty inefficient, especially as your project grows!
A lesser-known but better alternative is to use Django’s support for watchman. Support was added in Django 2.2, thanks to Tom Forbes in Ticket #27685.
Watchman is an efficient file watcher open sourced by Facebook. It works by receiving file change notifications from your operating system and bundling them together. When nothing is changing, it doesn’t need to do any work - saving processing power and consequently laptop battery life. And when something changes, Django gets notified about it in milliseconds. Watchman is also smart with changes, batching them together, and integrating with Git to wait for operations like switching branches to finish.
Setup
It’s just a few steps to get Django using watchman, covered in a couple paragraphs in the runserver
documentation. Here I’ll cover the exact commands it needed me to run to set it up for DB Buddy on macOS, which is a pretty vanilla Django 3.1 project.
First, I installed watchman, following the installation instructions on its site. On macOS, this meant just running brew install watchman
.
Second, I installed pywatchman
, the library Django uses to interface with watchman. I use pip-compile
from pip-tools to manage my requirements, so first I added pywatchman
to my requirements.in
:
diff --git a/requirements.in b/requirements.in
index c376421..4407458 100644
--- a/requirements.in
+++ b/requirements.in
@@ -28,2 +28,3 @@ pytest-xdist
python-dotenv
+pywatchman
requests
I then ran pip-compile
to pin the current version of pywatchman
in my requirements.txt
, resulting in this change:
diff --git a/requirements.txt b/requirements.txt
index 95002f9..5d9073c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -155,2 +155,4 @@ pytz==2020.5
# tzlocal
+pywatchman==1.4.1
+ # via -r requirements.in
requests-mock==1.8.0
I then ran pip install -r requirements.txt
to install the compiled requirements file.
Third, I set up a watchman configuration file in order to ignore my node_modules
directory. The Django documentation hints that ignoring non-Python directories like this may be sensible to reduce load. My project has a relatively minimal JavaScript setup, but that still amounts to 14,303 files in the node_modules
directory, all of which don’t need watching.
I followed the watchman configuration documentation and created a .watchmanconfig
file next to my manage.py
containing:
{
"ignore_dirs": ["node_modules"]
}
Fourth, I restarted runserver
. Django confirmed it’s using watchman by listing WatchmanReloader
as the file watcher on the first line:
$ python manage.py runserver
Watching for file changes with WatchmanReloader
Performing system checks...
System check identified no issues (1 silenced).
January 20, 2021 - 07:49:15
Django version 3.1.5, using settings 'db_buddy.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Great!
Benchmark
I followed this up with a quick before/after comparison of CPU usage, whilst the server was completely idle and not serving any requests.
Using StatReloader
, runserver
took about ~1.6% of a CPU core.
With WatchmanReloader
, this dropped to 0%, with the watchman
process also showing 0%. That’s fantastic!
This project is also relatively small at the moment, with only 50 Python files and a few dependencies. Since the StatReloader
’s work is proportional to the number of files, the difference will only be more stark on larger projects.
Fin
If you’re interested in learning more about the implementation of Django’s watchman integration, check out Tom Forbes’ EuroPython talk.
May your edit cycle be ever faster and more efficient,
—Adam
Make your development more pleasant with Boost Your Django DX.
One summary email a week, no spam, I pinky promise.
Related posts:
- Introducing django-linear-migrations
- Prevent Unintended Data Modification With django-read-only
- Bonus Django Documentation Sites
Tags: django