Python Type Hints - How to Enable Postponed Evaluation With __future__.annotations

What is postponed evaluation?

When Python introduced type hints, the team decided type hint expressions should execute at runtime. This allowed flexibility and experimentation with the syntax.

Unfortunately it also meant that expressions have to follow Python’s rules about variable scope. This made forward references, to classes that aren’t yet defined, problematic.

For example, take this class that can copy itself:

from copy import deepcopy


class Widget:
    ...

    def copy(self) -> Widget:
        return deepcopy(self)

The -> Widget return type hint is a forward reference, as the Widget class is not defined until the end of its class statement. With the old behaviour, Python tries to evaluate this reference, causing a NameError:

$ python example.py
Traceback (most recent call last):
  File "/.../example.py", line 4, in <module>
    class Widget:
  File "/.../example.py", line 7, in Widget
    def copy(self) -> Widget:
NameError: name 'Widget' is not defined

Type checkers introduced a workaround: wrapping type hints in quotes, to turn them into strings. Any such strings are then evaluated at the end of the file, by the type checker only. Python’s runtime sees them only as strings. For example:

from copy import deepcopy


class Widget:
    ...

    def copy(self) -> "Widget":
        return deepcopy(self)

This code runs successfully and passes type checking.

PEP 563 defined the new postponed evaluation behaviour. This moves all type hints to follow this pattern, without requiring the string syntax.

We can opt into postponed evaluation in a file by adding from __future__ import annotations at the top. In our example:

from __future__ import annotations

from copy import deepcopy


class Widget:
    ...

    def copy(self) -> Widget:
        return deepcopy(self)

Python no longer interprets the type hints at runtime, so it doesn’t hit the NameError. And type checkers can still work with the type hints as they evaluate them after reading the whole file.

Another benefit is that type hints are kept in string form at runtime, unless they’re explicitly used. This provides some nice memory savings, valuable on larger projects.

Update (2022-10-18): Updated the below to reflect the decision in Python 3.11.

Postponed evaluation is opt-in from Python 3.7, and was originally scheduled to become the default in Python 3.10. This was deferred until Python 3.11, and then again until… maybe never. The Python 3.11 release notes say that the feature “may not be the future”. This links to the steering council’s post, which notes the outstanding backwards-incompatible issues, and asks for input.

Compatibility issues

The compatibility issues came from Python libraries that use type hints at runtime. Python provides the typing.get_type_hints() function to fetch the type hints for a function or class at runtime. Unfortunately due to Python’s scoping rules it doesn’t always work with postponed evaluation.

For example, take this code:

import typing


def make_author():
    class Book:
        pass

    class Author:
        book: Book

    return Author


Author = make_author()
print(typing.get_type_hints(Author))

The Author class has a reference to Book, which only exists inside the locals of the call to make_author().

When we run the code with the old behaviour, non-postponed evaluation, we see the dict of type hints for Author:

$ python example.py
{'book': <class '__main__.make_author.<locals>.Book'>}

But if we add from __future__ import annotations to activate the new behaviour, the code crashes:

$ python example.py
Traceback (most recent call last):
  File "/.../example.py", line 17, in <module>
    print(typing.get_type_hints(Author))
  File "/.../lib/python3.10/typing.py", line 1696, in get_type_hints
    value = _eval_type(value, base_globals, base_locals)
  File "/.../lib/python3.10/typing.py", line 307, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "/.../lib/python3.10/typing.py", line 650, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
NameError: name 'Book' is not defined. Did you mean: 'bool'?

For now there is no fix in Python for this issue. Some libraries using runtime type hints have workarounds, such as Pydantic’s update_forward_refs() function.

Thankfully, use of such local-scoped forward references is rare. If it is stopping you adding from __future__ import annotations to a whole module, consider splitting off the problematic code into its own module.

Adding to all files

Consider adding from __future__ import annotations to all your files. This can be done automatically with isort.

You can set up isort in many ways, for example with the pre-commit framework. Once you have isort running, you can set its add_imports option to add the import. For example, here’s the configuration I’m using in my pyproject.toml:

[tool.isort]
profile = "black"
add_imports = "from __future__ import annotations"

After adding the import to all files, you can upgrade your syntax with pyupgrade. For files with from __future__ import annotations it automatically unquotes stringified type hints.

Fin

May you be __future__-proof,

—Adam