Sometimes error reporting loses details, and you don’t know where an error message comes from. In such cases, I have often found that the fastest way to track down the source is to search through the project and its virtualenv.
Investigating flaky tests is a dull necessity of testing. At least it is (in the best case) infrequent. Here are some shell commands you can use to automate steps in your investigations. These will work on (at least) bash and zsh.
CloudFormation’s stack drift detection feature is useful. It discovers ways your infrastructure that you beautifully set up with Infrastructure-as-Code has been fiddled with manually. Often this results from a “quick temporary fix” being applied manually on the web console at 2am, then forgotten about.
I maintain a few Python projects on Github + PyPI. I try to keep them as consistent and error-free as possible and thus running code quality tools (known as ‘linters’) such as flake8 and isort to ensure this happens. I also discovered the lesser known python setup.py check which has found a number of problems I otherwise wouldn’t have found until release time - highly recommended!