How to Add Database Modifications Beyond Migrations to Your Django Project2019-08-07
On several Django projects I’ve worked on, there has been a requirement for performing database modifications beyond Django migrations. For example:
- Managing stored procedures
- Managing check constraints, which weren’t supported before Django 2.2
- Importing static data from a file
- Recording migration operations in a log
Let’s look at three approaches to extending Django to do this as neatly as possible.
1. Use Django migrations
Often I find developers have only been taught how to use Django migrations for model operations. They might know some SQL, but since they haven’t used it within Django, they assume migrations can’t use SQL directly. They can! Many of the uses of custom migration-style SQL code that I’ve seen could be better implemented within migrations.
Let’s take a look at an example we’ll use for the rest of the article. Imagine you’re running a Django version before 2.2 that doesn’t support database check constraints. You might add one to a database table with this SQL:
This constraint will make the database raise an error for any rows added or updated in the
myapp_book table that have a percentage of lovers and haters not equal to 100.
With Django’s default table naming, the table
myapp_book would be created for a model called
Book inside the app
We’ll use those names in this article too.
The SQL above could be run using
Instead of doing those though, we can turn it into a migration.
First, you can create a new migration with:
You can then edit that migration file to use the
A little bonus step is to rename the migration file to something descriptive.
For example, instead of
0101_auto_20190715_1057.py we could call this
This helps a lot in the long term.
To improve the migration beyond our initial SQL, we can add the
This tells Django how to reverse the migration.
You’ll rarely need to reverse a migration, but when you do… you really do!
Be prepared, as I learned in the Scouts.
For our example, we can expand the migration operation to:
This will remove the check constraint when reversing the operation.
If the code you want to run is more complex, for example you want to run something on every model/table, you might use the
With it you can do pretty much anything - the power of Python!
It also supports a reverse option, which is worth adding - check out the docs.
This approach fits many use-cases, and it also helps provide a lifetime for any objects created by your custom SQL.
For example, if we wanted to drop our constraint at a later date, we could create a similar migration with
DROP CONSTRAINT as the forwards SQL.
One drawback is that it will only run once. This might not make sense if you have something that needs running frequently, as you might need to add a lot of operations.
2. Override the ‘migrate’ management command
This feels like a bit of a secret feature, given how few projects I’ve seen use it. However, I have found it handy on several occasions.
Django allows you to override management commands by adding another with the same name. You can override commands from Django core, or those in other apps. This is documented in “custom management commands”.
When running a command, Django searches through all the apps in
INSTALLED_APPS, and then the core.
The first place it finds a management command with the given name wins.
Thus, your apps can override any built-ins.
migrate and add your custom behaviour, you’ll want to create
myapp with the name of one of your apps).
Inside that file you can then subclass the built-in migrate command and add your own behaviour.
(The custom management commands documentation is likely helpful for writing your own command.)
Overriding works because all throughout Django’s code,
migrate is called as a command rather than imported directly.
This is done with the
So, if you’ve overridden
migrate, your new version is called instead of the one from core.
You can override any built-in command.
This approach is more flexible than using
RunSQL in migrations.
We can add any code we want before or after
migrate runs - or even “during” with a context manager.
The major drawback here is we can only override once, in a single app, so it could feel a bit clumsy if we have several app-specific extensions. However, for most projects, I’d recommend keeping it simple.
Having a single “project app” can work really well - I endorse the recommendation in Kristian Glass’ Unofficial FAQ. If you already have multiple apps, you can make one the “core,” have it contain your custom migrate, and then import code from the others. This will be just fine.
That said, sometimes we we want looser coupling, for example when creating third party packages. So let’s look at a final approach.
3. Adding a
post_migrate signal handler
This approach is slightly more advanced again. It uses Django’s signals, which have a mixed reputation due to their “action at a distance.”
To run some extra code at that point, write it as a signal handler.
Registering a signal handler is best done in an
AppConfig.ready() method, which Django will call at initialization time.
For an example, let’s look at Django’s contenttypes framework.
This is included as
It uses a
post_migrate signal handler to create one
ContentType model instance for each model.
create_contenttypes handler is registered in its
AppConfig.ready() like so:
The handler is defined in the app’s
This is not the most descriptive filename to contain a signal handler - I’d normally use a
handlers.py within the app.
The contenttypes framework has it there for historical reasons.
In Django 2.2.3,
create_contenttypes is defined like so:
There is quite a lot of logic here. We can ignore most of it right now though, as it’s use-case specific, but you can get the gist by reading it.
The interesting thing to look at is the function signature.
Signal handlers are only called with keyword arguments.
For forwards compatibility, they should accept any extras at the end in
**kwargs, so that unrecognized arguments added by the sender don’t break the handler - more loose coupling.
The arguments listed here are all as per the
app_configis the current
AppConfig- the signal is sent once for each app.
verbosityis the current logging level.
interactivetells us if it’s safe to ask the user for input.
usingis the database connection alias, which will vary from
DEFAULT_DB_ALIASwhen using multiple databases.
appsis an application registry containing the specific state of all models after the migrations have run. Because the user might not have run every migration available, this should be used to access model classes, instead of direct imports.
For writing most custom handlers, I think two of these are the most useful.
app_config can be used to restrict your handler to only run for a specific app or set of apps.
You can also do this with the
sender argument to
using is worth passing through for any database operations you use, even if for future proofing.
Even if your project uses a single database now, it might not in the future, so you should make sure you operate on the same connection that was migrated.
As we’ve discussed, the benefit of this approach is the looser coupling. If you’re writing a third party package, this is probably the way to go, as it reduces the amount of things that users need to install. Signals do require caution, but since we’ve seen Django itself uses this in a contrib app, it’s a sanctioned use.
I hope this article helps you find the right approach for your project,
Are your Django project's tests slow? Read Speed Up Your Django Tests now!
One summary email a week, no spam, I pinky promise.
- Getting a Django Application to 100% Test Coverage
- How to Score A+ for Security Headers on Your Django Website
- "Create Table As Select" in Django
© 2020 All rights reserved.