Python Type Hints - How to Enable Postponed Evaluation With __future__.annotations

2021-05-15 from future import ant-onations

Python 3.7 added a new “postponed evaluation” mode for type hints. This is opt-in for now, and may become the default in Python 3.11. In this post we’ll cover how it changes the behaviour of type hints, how to activate it, and the outstanding problems.

What is postponed evaluation?

When Python introduced type hints, the team decided type hint expressions should execute at runtime. This allowed flexibility and experimentation with the syntax.

Unfortunately it also meant that expressions have to follow Python’s rules about variable scope. This made forward references, to classes that aren’t yet defined, problematic.

For example, take this class that can copy itself:

from copy import deepcopy


class Widget:
    ...

    def copy(self) -> Widget:
        return deepcopy(self)

The -> Widget return type hint is a forward reference, as the Widget class is not defined until the end of its class statement. With the old behaviour, Python tries to evaluate this reference, causing a NameError:

$ python example.py
Traceback (most recent call last):
  File "/.../example.py", line 4, in <module>
    class Widget:
  File "/.../example.py", line 7, in Widget
    def copy(self) -> Widget:
NameError: name 'Widget' is not defined

Type checkers introduced a workaround: wrapping type hints in quotes, to turn them into strings. Any such strings are then evaluated at the end of the file, by the type checker only. Python’s runtime sees them only as strings. For example:

from copy import deepcopy


class Widget:
    ...

    def copy(self) -> "Widget":
        return deepcopy(self)

This code runs successfully and passes type checking.

PEP 563 defined the new postponed evaluation behaviour. This moves all type hints to follow this pattern, without requiring the string syntax.

We can opt into postponed evaluation in a file by adding from __future__ import annotations at the top. In our example:

from __future__ import annotations

from copy import deepcopy


class Widget:
    ...

    def copy(self) -> Widget:
        return deepcopy(self)

Python no longer interprets the type hints at runtime, so it doesn’t hit the NameError. And type checkers can still work with the type hints as they evaluate them after reading the whole file.

Postponed evaluation is opt-in from Python 3.7, and may become the default in Python 3.11. Originally the PEP planned to make this the default from Python 3.10. But, due to the compatibility problems, the Python Steering Council decided to postpone the change for one more release. Exactly what will change in Python 3.11 is still up for debate, but it be one step closer to turning on postponed evaluation by default.

Compatibility issues

The compatibility issues came from Python libraries that use type hints at runtime. Python provides the typing.get_type_hints() function to fetch the type hints for a function or class at runtime. Unfortunately due to Python’s scoping rules it doesn’t always work with postponed evaluation.

For example, take this code:

import typing


def make_author():
    class Book:
        pass

    class Author:
        book: Book

    return Author


Author = make_author()
print(typing.get_type_hints(Author))

The Author class has a reference to Book, which only exists inside the locals of the call to make_author().

When we run the code with the old behaviour, non-postponed evaluation, we see the dict of type hints for Author:

$ python example.py
{'book': <class '__main__.make_author.<locals>.Book'>}

But if we add from __future__ import annotations to activate the new behaviour, the code crashes:

$ python example.py
Traceback (most recent call last):
  File "/.../example.py", line 17, in <module>
    print(typing.get_type_hints(Author))
  File "/.../lib/python3.10/typing.py", line 1696, in get_type_hints
    value = _eval_type(value, base_globals, base_locals)
  File "/.../lib/python3.10/typing.py", line 307, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "/.../lib/python3.10/typing.py", line 650, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
NameError: name 'Book' is not defined. Did you mean: 'bool'?

For now there is no fix in Python for this issue. Some libraries using runtime type hints have workarounds, such as Pydantic’s `update_forward_refs() function.

Thankfully, use of such local-scoped forward references is rare. If it is stopping you adding from __future__ import annotations to a whole module, consider splitting off the problematic code into its own module.

Adding to all files

Assuming you’re not on Python 3.11 (unreleased at time of writing), consider adding from __future__ import annotations to all your files. This can be done automatically with isort.

You can set up isort in many ways, for example with the pre-commit framework. Once you have isort running, you can set its add_imports option to add the import. For example, here’s the configuration I’m using in my pyproject.toml:

[tool.isort]
profile = "black"
add_imports = "from __future__ import annotations"

(If you are on Python 3.11, you can swap to the remove_imports option instead!)

After adding the import to all files, also check out pyupgrade, which can upgrade your syntax. For files with from __future__ import annotations it automatically unquotes stringified type hints.

Fin

May you be __future__-proof,

—Adam


Want better tests? Check out my book Speed Up Your Django Tests which teaches you to write faster, more accurate tests.


Subscribe via RSS, Twitter, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: mypy, python