Better Python Decorators with wrapt2020-07-02
A Python decorator wraps a target function with another wrapper function. This wrapper function can add any behavior you might want. For example, it can track execution times, redefine how the wrapped function runs, or modify return values.
As a concrete example, the standard library’s
functools.lru_cache() decorator wraps a target function to add caching behavior.
If you call a wrapped function twice with the same arguments, the second call returns a cached value rather than re-executing the potentially costly inner function.
Scout’s Python integration uses decorators extensively for instrumenting third party libraries. This instrumentation allows us to measure execution time and help you find hot spots in your application.
Our instrumentation uses built-in extension mechanisms where possible, such as Django’s database instrumentation.
But often libraries have no such mechanisms, so we resort to wrapping third party libraries’ functions with our own decorators.
For example, we instrument
Template.render() function with a decorator to measure template rendering time.
We value correctness of our instrumentation a lot, so that we do not affect our users’ applications. It’s important that our decorators are as transparent as possible, so they do not break assumptions held by the library or application. Therefore, we create our decorators with the wrapt library by Graham Dumpleton.
wrapt makes writing correct, transparent decorators easy. If you are writing any kind of decorators in Python, you should consider using wrapt. Let’s look at how it helps the Scout integration.
Making a Decorator with the Function Closure Pattern
Imagine we have this simple function for generating HTML, imitating Jinja2’s
We’d like to instrument it with a decorator to print the execution time. We can create a decorator that does this like so:
This example uses the “function closure pattern” that is commonly used to create decorators.
instrument function is called with the wrapped function as
wrapped, and returns the
wrapper function can access the
wrapped function after
instrument() has completed through the function closure.
- Records the start in nanoseconds with
- Calls the wrapped function
wrapped, looked up in the function closure of the outer
- Records the end time.
- Prints a message about the execution time in nanoseconds.
- Returns the result from calling
You can imagine that in the Scout integration, rather than printing a message, we append details about the executed function to a log. This log is then shipped with extra details as a trace of a single web request. We aggregate this trace with those for similar requests and display the results in our GUI.
Using our Decorator
We can apply our decorator like so:
instrument() function returns our
wrapper() function, bound to the name
@ syntax is actually a shortcut added in Python 2.4.
We can expand it like so:
In this form, we can see that
render_html is effectively defined twice: first as a plain function, second as the return value of
instrument() returns our new
wrapper() function, it continues to be a function.
Decorators can return any Python object, although it normally doesn’t make sense to return something other than a function.
We can call our decorated
render_html() with a value for
We see the message from our wrapper printed, as well as the returned string.
So our decorator works, but unfortunately it’s not very transparent.
It replaces the inner
render_html() function with the
wrapper() function, and this causes several detectable differences.
We can see some of differences between the unwrapped and wrapped versions in their representations (through
If we execute only the function name, the interpreter shows us its representation.
@instrument, the function’s representation looks like this:
@instrument, the function representation is quite different:
We can see these differences in the decorated version:
- The function is no longer displayed as existing directly in
__main__, but within
instrument.<locals>, indicating it was created within that function closure.
- The name is no longer
- The argument specification is no longer
These pieces of the function representation are all included from some special attributes.
For example, the “qualified name” for the function is stored in its
__qualname__ attribute, which is where
<locals> comes from for the instrumented version:
The changes to these attributes make debugging harder, as it can be hard to tell what functions are referred to. They can also break tools that rely on introspection. “Transparent” decorators preserve these attributes so that the full details of the source function can be used in debugging and introspection.
Copying Attributes for Transparency
We can make our decorator a little bit more transparent by making it copy over
Now the wrapped function’s representation includes the correct name:
The argument specification is still wrong though, as well as several other attributes we haven’t looked at.
We can copy all of them by using the standard library function
We would use it in our decorator like so:
Now when we check the function representation, it’s the same as without the decorator:
This is great. If the function representation was all we cared about for transparency, we could stop here. Unfortunately, there are some other differences that might affect certain applications.
Prefect Transparency with Wrapt
The author of the wrapt library that we use, Graham Dumpleton, wrote a series of blog posts on issues with decorators. It starts with the post How you implemented your Python decorator is wrong, which identifies these issues:
- Preservation of function
- Preservation of function argument specification.
- Preservation of ability to get function source code.
- Ability to apply decorators on top of other decorators that are implemented as descriptors.
We solved the first two by using
functools.update_wrapper(), as it copies over the relevant attributes.
However, the second two are much more difficult to solve, and we’re glad wrapt does them for us.
They’re a bit too much to describe here, but if you’re interested, you should check out the blog post series.
We can reimplement our above decorator using wrapt like so:
wrapt.decorator is a decorator for creating decorators.
It takes our
instrument() function, and turns it into a fully-featured transparent decorator.
It saves us writing two levels of function, allowing us to focus solely on our wrapper.
Our instrument function takes four arguments that wrapt passes us on each call:
wrapped, the wrapped function. If we supported decorating classes, this could also be the wrapped class.
instance, the wrapped object instance. In this case we only support wrapping plain functions, rather than methods, so this will always be
args, a tuple of all the positional arguments used in the current call.
kwargs, a dictionary of all the keyword arguments used in the current call.
The body of
instrument is similar to the previous implementation, with
wrapped(*args, **kwargs) calling the wrapped function.
The wrapped function works exactly the same as our previous implementation, and has the same, transparent representation:
And That’s a Wrapt
wrapt also comes with some other neat features:
- You can enable or disable decorators temporarily. This can even be done dynamically on a per-call basis.
- You can use
wrapt.decoratorto implement universal decorators that cover classes, functions, methods, and class methods.
- You can use the underlying object proxy class independently for custom transparent proxying of any Python object.
- The wrapt core implementation is in a C extension, reducing its overhead drastically.
To learn more, check out its documentation, and the series of blog posts.
You may also be interested to see our usage in the Scout Python integration.
A good starting point is the instrumentation of Jinja2’s
There’s also the pull request where we moved to wrapt.
May your code be well decorated,
Working on a Django project? Check out my book Speed Up Your Django Tests which covers loads of best practices so you can write faster, more accurate tests.
One summary email a week, no spam, I pinky promise.
© 2020 All rights reserved.