Profilers measure the performance of a whole program to identify where most of the time is spent. But once you’ve found a target function, re-profiling the whole program to see if your changes helped can be slow and cumbersome. The profiler introduces overhead to execution and you have to pick out the stats for the one function you care about from the report. I have often gone through this loop while optimizing client or open source projects, such as when I optimized Django’s system checks framework (previous post).
When testing code that outputs to the terminal through either standard out (stdout) or standard error (stderr), you might want to capture that output and make assertions on it. To do so, use contextlib.redirect_stdout() and contextlib.redirect_stderr() to redirect the respective output streams to in-memory buffers that you can then inspect and assert on.
Sometimes, it’s useful to branch the behaviour of your code based on the version of a package that you have installed. This may be to support an upgrade in your project, or for your own package to support different versions of a dependency.
If you’ve written a Python script that outputs a lot of data, then piped that output into another command that only reads part of it, you might have encountered a BrokenPipeError. For example, take this script:
A neat testing pattern is writing common tests in a base class and then applying them to multiple objects through subclassing. Doing so can help you test smarter and cover more code with less boilerplate.
Python comes with two built-in profilers for measuring the performance of your code: cProfile and profile. They have the same API, but cProfile is a C extension, while profile is implemented in Python. You nearly always want to use cProfile, as it’s faster and doesn’t skew measurements as much.
pre-commit is my favourite Git-integrated “run things on commit” tool. It acts as a kind of package manager, installing tools as necessary from their Git repositories. This makes it fairly easy to set up: all you need to install is pre-commit itself, and it takes things from there.
In Python, a mixin class is a class that is not intended to be used directly, but instead “mixed in” to other classes through multiple inheritance. Mixins are not really a language feature but more of a conventional pattern allowed by Python’s multiple inheritance rules. Unfortunately, adding type hints to mixin classes can be tricky because they implicitly require subclasses to fit a certain shape.
Some Python packages require native libraries to be installed on your system when you install them. For example, mysqlclient requires libssl and libcrypto. If those libraries are missing at install time, clang, the C compiler, fails with a message like:
A few days ago, I blogged about debugging unraisable exceptions with Rich. Here’s a sequel on testing that some block of code doesn’t trigger any unraisable exceptions.
Python 3.12 introduced sys.monitoring, a new framework for “monitoring” tools like debuggers and profilers to hook into. It provides fine-grained control so tools can listen only to certain events on specific lines of code. The framework came from PEP 669, thanks to Mark Shannon of the Faster CPython team.
Sometimes, tests need temporary files or directories. You can do this in Python’s unittest with the standard library tempfile module. Let’s look at some recipes to do so within individual tests and setUp().
tracemalloc is Python’s standard library module for tracking memory allocations. It has many bells and whistles for detailed analysis, allowing you to slice allocations by file and line or compare snapshots. But for simple purposes, displaying the total memory allocated is sufficient, which is what this recipe does:
Django and other frameworks often allow you to configure classes, functions, or modules to import using strings. To do the same in your own code, use pkgutil.resolve_name() from the standard library (added in Python 3.9):
Sometimes, you want to test a function that uses an inner import and mock that imported object. For example, say you wanted to mock dig() in this example:
Copy-paste-tweaking library code feels like a dirty but inevitable programming practice. Often driven by deadlines or other constraints, it seems all projects end up with something copy-pasted in and tweaked for one specific use case.
Many terminals and text editors support what I’ll call “line number paths” of the form <filename>:<lineno>. Opening that path, whether by clicking or passing to a CLI, opens the given file at that line.
Earlier this week, I made another podcast appearance on The Python Show, episode 22: Git and Django with Adam Johnson. As fellow authors, Mike and I talked a lot about the writing process, on topics like:
I had the pleasure of returning to The Real Python podcast in last week’s episode 179, Improving Your Git Developer Experience in Python. It was great to catch up with host Christopher Bailey and chat about: