Python Type Hints - Three Somewhat Unexpected Uses of typing.Any in Python’s Standard Library
When we add type hints, we can find our desire for strictness in tension with Python’s flexibility. In this post we’ll explore three groups of functions in the standard library that I naïvely expected to use narrow types, but due to some edge cases, instead use
operator module provides wrapper functions for Python’s operators. For example,
operator.gt(a, b) wraps the “greater than” operator, so is equivalent to
a > b.
We normally expect operators to have well defined types. For example, comparison operators like
bool values, as stated by the syntax documentation.
But Python allows operator overloading, which makes it possible for custom operator functions to return an arbitrary types.
pathlib.Path does this, to great effect, using the division operator
/ for concatenation.
This flexibility forces the types for the
operator module functions to accept and return
Any. Here are the current typeshed stubs for the comparison functions in
def lt(__a: Any, __b: Any) -> Any: ... def le(__a: Any, __b: Any) -> Any: ... def eq(__a: Any, __b: Any) -> Any: ... def ne(__a: Any, __b: Any) -> Any: ... def ge(__a: Any, __b: Any) -> Any: ... def gt(__a: Any, __b: Any) -> Any: ...
Given the flexibility of
operator, we might find it better to re-implement the functions with narrow types for our specific use cases. For example:
def gt(a: int, b: int) -> bool: return a > b
logging module is documented as taking string log messages:
The msg is the message format string, and the args are the arguments which are merged into msg using the string formatting operator.
So we could reasonably expect the type hints for
Logger.debug() and co. all define
msg as a
str. But in reality all the methods define
typeshed PR #1776 changed from
Any, and explains why. The core
Logger method uses
str(msg) to force
msg into a string, meaning it allows any type. But using a non-
str type may represent an mistake that makes the log message useless, which Guido van Rossum lamented on the typeshed PR:
In this particular case, I’m still sad to see the logging msg argument change from str to Any, because it will reduce opportunities to catch errors. In my experience *usually* this is a coding mistake.
JSON has a particularly limited set of types, which it would seem would translate well into a type hint for
json.loads(). There are four atomic types:
null- loaded as
- booleans - loaded as
- numbers - loaded as
- strings - loaded as
…and two container types:
- arrays - loaded as
- objects - loaded as
The container types can contain any of the atomic types or other containers.
This containers-can-contain-containers recursion is our first problem for representing JSON in type hints. We need to use recursive type hints, which unfortunately Mypy does nont currently support. If we try and define the JSON types recursively, like so:
from typing import Dict, List, Union _PlainJSON = Union[ None, bool, int, float, str, List["_PlainJSON"], Dict[str, "_PlainJSON"] ] JSON = Union[_PlainJSON, Dict[str, "JSON"], List["JSON"]]
…Mypy will report “possible cyclic definition” errors:
$ mypy example.py example.py:3: error: Cannot resolve name "_PlainJSON" (possible cyclic definition) example.py:4: error: Cannot resolve name "_PlainJSON" (possible cyclic definition) example.py:6: error: Cannot resolve name "JSON" (possible cyclic definition) Found 3 errors in 1 file (checked 1 source file)
Support for recursive types in Mypy is feasible, and is tracked in its Issue #731.
But, even if Mypy adds recursive type support,
json.loads() will still need to use a return type of
Any. This is, again, due to extra flexibility in its API.
json.loads() accepts several extra arguments that can be used to change JSON’s types to load into different Python types. Notably, the
cls argument allows complete replacement of the loading machinery, so we can have JSON parse into any type. Thus,
json.loads() will always require a return type of
Libraries for other formats, such as PyYAML have adopted the same pattern. Therefore they also use a return type of
We’ve seen that the pesky
Any type may be “hiding” in API’s that are normally well-typed, but offer some flexibility. We need to take care when using such functions.
As type hints spread through the Python ecosystem, we may see such API’s changed to allow stricter typing in the common cases. For example,
json.loads() could be split into two functions: one offering no flexibility with a well-defined return type, and one offering all the customization with a return type of
Learn how to make your tests run quickly in my book Speed Up Your Django Tests.
One summary email a week, no spam, I pinky promise.
- Python Type Hints - How to Debug Types With reveal_type()
- Python Type Hints - How to Enable Postponed Evaluation With __future__.annotations
- Python Type Hints - Use object instead of Any
Tags: mypy, python