Python Type Hints - Three Somewhat Unexpected Uses of typing.Any in Python’s Standard Library

2021-06-14 “If all you have is an Any, everything looks like an Any.”

When we add type hints, we can find our desire for strictness in tension with Python’s flexibility. In this post we’ll explore three groups of functions in the standard library that I naïvely expected to use narrow types, but due to some edge cases, instead use Any.

1. operator functions

The operator module provides wrapper functions for Python’s operators. For example, operator.gt(a, b) wraps the “greater than” operator, so is equivalent to a > b.

We normally expect operators to have well defined types. For example, comparison operators like > return bool values, as stated by the syntax documentation.

But Python allows operator overloading, which makes it possible for custom operator functions to return an arbitrary types. pathlib.Path does this, to great effect, using the division operator / for concatenation.

This flexibility forces the types for the operator module functions to accept and return Any. Here are the current typeshed stubs for the comparison functions in operator:

def lt(__a: Any, __b: Any) -> Any: ...
def le(__a: Any, __b: Any) -> Any: ...
def eq(__a: Any, __b: Any) -> Any: ...
def ne(__a: Any, __b: Any) -> Any: ...
def ge(__a: Any, __b: Any) -> Any: ...
def gt(__a: Any, __b: Any) -> Any: ...

Given the flexibility of operator, we might find it better to re-implement the functions with narrow types for our specific use cases. For example:

def gt(a: int, b: int) -> bool:
    return a > b

2. The logging module

Python’s logging module is documented as taking string log messages:

The msg is the message format string, and the args are the arguments which are merged into msg using the string formatting operator.

So we could reasonably expect the type hints for Logger.debug() and co. all define msg as a str. But in reality all the methods define msg as Any. Why?

typeshed PR #1776 changed from str to Any, and explains why. The core Logger method uses str(msg) to force msg into a string, meaning it allows any type. But using a non-str type may represent an mistake that makes the log message useless, which Guido van Rossum lamented on the typeshed PR:

In this particular case, I’m still sad to see the logging msg argument change from str to Any, because it will reduce opportunities to catch errors. In my experience *usually* this is a coding mistake.

Aw, shucks.

3. json.loads() and friends

JSON has a particularly limited set of types, which it would seem would translate well into a type hint for json.loads(). There are four atomic types:

…and two container types:

The container types can contain any of the atomic types or other containers.

This containers-can-contain-containers recursion is our first problem for representing JSON in type hints. We need to use recursive type hints, which unfortunately Mypy does nont currently support. If we try and define the JSON types recursively, like so:

from typing import Dict, List, Union

_PlainJSON = Union[
    None, bool, int, float, str, List["_PlainJSON"], Dict[str, "_PlainJSON"]
]
JSON = Union[_PlainJSON, Dict[str, "JSON"], List["JSON"]]

…Mypy will report “possible cyclic definition” errors:

$ mypy example.py
example.py:3: error: Cannot resolve name "_PlainJSON" (possible cyclic definition)
example.py:4: error: Cannot resolve name "_PlainJSON" (possible cyclic definition)
example.py:6: error: Cannot resolve name "JSON" (possible cyclic definition)
Found 3 errors in 1 file (checked 1 source file)

Support for recursive types in Mypy is feasible, and is tracked in its Issue #731.

But, even if Mypy adds recursive type support, json.loads() will still need to use a return type of Any. This is, again, due to extra flexibility in its API.

json.loads() accepts several extra arguments that can be used to change JSON’s types to load into different Python types. Notably, the cls argument allows complete replacement of the loading machinery, so we can have JSON parse into any type. Thus, json.loads() will always require a return type of Any.

Libraries for other formats, such as PyYAML have adopted the same pattern. Therefore they also use a return type of Any.

Conclusion

We’ve seen that the pesky Any type may be “hiding” in API’s that are normally well-typed, but offer some flexibility. We need to take care when using such functions.

As type hints spread through the Python ecosystem, we may see such API’s changed to allow stricter typing in the common cases. For example, json.loads() could be split into two functions: one offering no flexibility with a well-defined return type, and one offering all the customization with a return type of Any.

Fin

May your type hints take you Anywhere you want to go,

—Adam


🎉 My book Speed Up Your Django Tests is now up to date for Django 3.2. 🎉
Buy now on Gumroad


Subscribe via RSS, Twitter, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: mypy, python