Django: django-upgrade 1.32.0 out now, with 44 AI-assisted bug fixes

Bad url() to path() conversions

The django_urls fixer converts old url() calls, with their regular expression patterns, into path() calls with the newer route syntax, where possible.

Claude found two cases where the “where possible” condition was too optimistic.

First, literal angle brackets. In a regular expression, < and > are literal characters, but in path() route syntax they declare parameters. The fixer copied them through unchanged:

-    url(r"^go/<page>/$", views.redirect_angle),
+    path("go/<page>/", views.redirect_angle),

The old pattern matched only the exact URL /go/<page>/, angle brackets included. The new route matches /go/anything/ and passes page as a keyword argument to the view, which likely isn’t expecting it. And if the bracketed text isn’t a valid Python identifier, like <not-a-name>, Django instead raises ImproperlyConfigured at startup. PR #715 made the fixer skip such patterns.

Second, unescaped dots. In a regular expression, a bare . matches any character, whilst \. matches only a literal dot. The fixer treated both the same, converting:

-    url(r"^report.pdf$", views.report),
+    path("report.pdf", views.report),

The old pattern also matched URLs like /reportxpdf, so the conversion silently narrowed which URLs the pattern matches. Most such patterns contain a “latent bug” where the author meant \., but it’s not django-upgrade’s place to change behaviour. PR #710 made the fixer leave patterns with unescaped dots alone.

These two bug fixes have hopefully closed a loop for me on a client project. Last year, I tried applying django-upgrade to a large client project, and it failed some tests. In my investigation, I cut down the fixers being applied to a short list, including django_urls, and still some tests failed. I reached the suspicion that some URLs were being converted incorrectly, but I ran out of time to properly investigate. Now a bot has found and fixed these bugs without me even trying, I’ll be trying that upgrade again!

Escaping backwards

Django 3.0 deprecated django.utils.text.unescape_entities() in favour of Python’s html.unescape().

Back in version 1.2.0 (2021), I added the unescape_entities fixer to django-upgrade to rewrite calls to the new function. But I accidentally made it rewrite calls to the inverse function, html.escape() instead of html.unescape():

-from django.utils.text import unescape_entities
+import html

-text = unescape_entities(raw)
+text = html.escape(raw)

Woops! You can see how the results would vary:

>>> html.unescape("Tom & Jerry")
'Tom & Jerry'
>>> html.escape("Tom & Jerry")
'Tom & Jerry'

😬

PR #684 was the bug fix to correctly rewrite code to use html.unescape().

This fixer survived nearly five years of use, perhaps through a combination of few projects activating it, no one noticing when it broke their code, and users potentially working around the issue by disabling the fixer. I’m glad Claude could spot the obvious error.

TestCase.multi_db = False fixer blocking all database queries

Django 2.2 replaced the test case attributes allow_database_queries and multi_db with databases.

django-upgrade rewrites those old attributes to the new one, but it mapped multi_db = False to an empty list:

 class OrderTests(TestCase):
-    multi_db = False
+    databases = []

That looks sensible at first glance, but it’s wrong. Under Django’s deprecation shim, multi_db = False still allowed queries against the default database. The rewritten databases = [] blocks queries against all databases, breaking previously-working tests with DatabaseOperationForbidden errors.

PR #727 corrected the mapping:

 class OrderTests(TestCase):
-    multi_db = False
+    databases = ["default"]

This is another bug I introduced in version 1.2.0. I guess no one hit this code path, since it would trigger an obvious test case breakage. But with open source, it is hard to know how many folks will actually make a bug report.

Bar height++

These bug fixes are a subset of the 44 in the release—see the changelog for the full list. There are definitely some more bugs lurking, but for now I’m out of time and energy for django-upgrade. I even left some harder-to-review bug fixes in draft PRs for my next pass at the project.

But yeah, since bugs like these are now fairly cheap to find, I hope that the software quality bar goes up. Applying fixes still requires some vigilance in review and checks from deterministic tools like linters, but good projects generally already apply such tools. And while it can be hard to trust LLMs to build features, where they often large piles of code, this genre of small, targeted bug fixes are an easy win.

Try my prompt on your own project. Here it is again:

Find and fix bugs

Fin

May robot eyeballs make all your bugs shallow,

—Adam