How to search your virtualenv for mystery error messages
Sometimes error reporting loses details, and you don’t know where an error message comes from. In such cases, I have often found that the fastest way to track down the source is to search through the project and its virtualenv.
ripgrep is the best tool for such seaches. It’s a much-much faster version of classic grep, with bonus features.
In this post we’ll look at how to search your virtualenv with ripgrep, a real life example, and how to do the same with plain grep.
How to search
By default, ripgrep obeys .gitignore, and ignores hidden files and folders. In order to include yoru virtualenv for search, you’ll typically need to lift those restrictions with the -uu double option, short for --unrestricted (twice).
To search your project and the virtualenv contained within, you can use:
$ rg -uu 'Error message'
You can restrict your search to only .py files with the -t option, short for --type:
$ rg -uu -tpy 'Error message'
If your virtualenv doesn’t live within your project, you can search both your project and virtualenv by specifying relevant paths:
$ rg -uu 'Error message' . ~/.virtualenvs/example
Neat.
An Ansible eggsample
A few months ago, I was debugging an issue where Ansible would quit with a mysterious message, and no other details:
ERROR! A worker was found in a dead state
🤔🤔🤔
In order to find where in Ansible this message was triggered, I ripgrepped for it:
$ rg -uu -tpy 'ERROR! A worker was found in a dead state'
…but unfortunately this first search turned up zero results.
I figured the ERROR! part may be part of a generic error message display function, so I dropped that for a second search:
$ rg -uu -tpy 'A worker was found in a dead state'
venv/lib/python3.10/site-packages/ansible/plugins/strategy/__init__.py
837: raise AnsibleError("A worker was found in a dead state")
863: raise AnsibleError("A worker was found in a dead state")
Aha! From these results, I could dig in to the issue further. In this case I temporarily modified Ansible’s code to display more information.
In general, if your first searsch has no matches, try searching for just a part of the error message. Error logging often adds prefixes or suffixes, and message strings in code can be broken across multiple lines.
Alternatively with grep -r
If you can’t use ripgrep, plain old grep works as well. It doesn’t have any default filtering, so to search all files in the current directory, run it with -r for recursive:
$ grep -r 'Error message'
To filter to just .py files:
$ grep -r --include '*.py' 'Error message'
And if your virtualenv doesn’t live within your project, you can again specify multiple paths:
$ grep -r --include '*.py' 'Error message' . ~/.virtualenvs/example
Allllrighty then.
Beware that grep is much slower than ripgrep. On my machine, ripgrep is nearly ten times faster for an equivalent search.
Fin
For another example, see Anže Pečar’s blog post on searching through his virtualenv for vendored copies of six.
May your debugging adventures be fun and no longer than necessary,
—Adam
Improve your Django develompent experience with my new book.
One summary email a week, no spam, I pinky promise.
Related posts:
Tags: commandline