A Problem with Duplicated Mutable Constants

Hey! You’re just like me!

Here’s a small problem I’ve seen where several modules share versions of the same “constant” variable. It came up in the context of a Django project with multiple settings files, but it could happen in different contexts.

Imagine you have two submodules defining API_CONFIG as a “constant” dictinoary. The development submodule should copy the value in base, but use a different value for the "rate_limit" key. You have example/base.py which looks like:

API_CONFIG = {
    # ...
    "rate_limit": "10/m",
    # ...
}

…and example/development.py:

from example.base import API_CONFIG

API_CONFIG["rate_limit"] = "100/m"

By importing from base, the development module doesn’t need to completely redefine API_CONFIG. Great - the redundant repetition is reduced. But can you see the flaw in this approach?

The problem is that API_CONFIG is the same dict in both modules. The change in development “leaks” back to base:

In [1]: from example import base

In [2]: base.API_CONFIG["rate_limit"]
Out[2]: '10/m'

In [3]: from example import development

In [4]: development.API_CONFIG["rate_limit"]
Out[4]: '100/m'

In [5]: base.API_CONFIG["rate_limit"]
Out[5]: '100/m'

Eek!

This happens because Python variables are only names for underlying objects. Pointing another name to the same variable does not create a copy. If this is surprising, Ned Batchelder’s Python Names and Values is a great explainer.

The shared value might not always manifest as a problem. In the context of Django, it’s only possible to activate one settings file per process, so if you activated “development” the values in “base” won’t (normally) be needed. But it could be a problem if you ever need to use both modules, for example whilst debugging or testing.

Let’s look at a couple of ways of solving this problem.

Deep Copies

A simple technique here is to fully copy mutable objects before applying changes. Python lets you do this with copy.deepcopy(). In example/development.py, this looks like:

from copy import deepcopy

from example.base import API_CONFIG

API_CONFIG = deepcopy(API_CONFIG)
API_CONFIG["rate_limit"] = "100/m"

(Note that copy.copy() or dict.copy() won’t work. This is because they only produce shallow copies, which can share nested values.)

Immutable Values

A second solution is to switch to immutable data types. This way changes aren’t even possible to the value. For dicts, you can emulate an immutable dictionary with types.MappingProxyType, as I previous covered.

Using this in example/base.py would look like:

from types import MappingProxyType

API_CONFIG = MappingProxyType(
    {
        # ...
        "rate_limit": "10/m",
        # ...
    }
)

Then example/development.py can create a copy with changes:

from types import MappingProxyType

from example.base import API_CONFIG

API_CONFIG = MappingProxyType(
    API_CONFIG
    | {
        "rate_limit": "100/m",
    }
)

(Using Python 3.9’s dict merge operator.)

Neat.

Sorted

With either version of the above, the two modules preserve their separate values:

In [1]: from example import base

In [2]: base.API_CONFIG["rate_limit"]
Out[2]: '10/m'

In [3]: from example import development

In [4]: development.API_CONFIG["rate_limit"]
Out[4]: '100/m'

In [5]: base.API_CONFIG["rate_limit"]
Out[5]: '10/m'

👍

Fin

May your constants be constant,

—Adam


If your Django project’s long test runs bore you, I wrote a book that can help.


Subscribe via RSS, Twitter, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: ,