My plugin pytest-randomly was recently moved into the pytest-dev organization on GitHub, making it a bit “more official” as a pytest plugin. Thanks to Bruno Oliveira for suggesting it, Florian Bruhin and Bruno for approving it on the pytest-dev mailing list, and Gordon Wrigley for helping with its development.
In celebration I thought I’d explain a bit more of the background behind it. pytest-randomly really combines two functions:
- Controlling the random seed between test runs, which is useful when using a tool like Factory Boy to generate test data. By allowing the same seed to be used again, failures can be debugged. See more in my blog post on it.
- Reordering tests randomly, to discourage order-dependency, which can be common with certain fixture patterns touching global state like a database
For YPlan, we needed random seed control first. We added Factory Boy to shrink the test code needed to set up Django model instances, and to get more value from the tests by covering a wider range of cases between runs. We were using nose at the time, and implemented a plugin to reset the seed at the start of each test and a flag to control in just a few lines of code.
Later on we added the test reordering code, due to problems with Django’s
setUpTestData. This hook allows you to perform setup once at the start of all
the tests in a test class, and keep it between the individual test functions
with rollback at the database layer. The benefits can be huge as you can
perform the same setup for a bunch of tests once instead of tens of times. The
problem we encountered was mismatch between python and the database. Consider
The problem is that under the written test ordering, when
test_2 runs its
self.book object will be inconsistent between python and the database. The
database has been rolled back to the state from
setUpTestData, but on the
self.book.name will still be
'The Empire Striketh Back' -
there is no process to undo the changes from
test_1 in Python. When this
causes test failures it’s a bad case of action-at-a-distance, with state
leaking between tests.
This can be avoided by never caching model instances on the test cases, or
copy.deepcopy()ing them before modifying them inside a test function,
but both are easy to forget. And there is always other global state that can be
modified and not reset, like the contents of any module (yay Python 😁).
Order randomization is a way of protecting against this problem. If tests leak state and pass on one run, it’s unlikely they’ll pass on the next run as the order will be re-randomized.
We implemented randomization of order on nose by taking some existing code from ‘Massive Wombat’ (?) on Google Code and Nick Loadholtes on GitHub, merging it with our random seed reset plugin, and tidying it up to work with all the kinds of tests we had. After a while I had the chance to open source it as nose-randomly. Naturally, when we moved to pytest, this was rebuilt as pytest-randomly 👌
The main development on pytest-randomly since has been to make it reset the random generators used by some popular libraries, and to make it work with all the kinds of test items that pytest supports (sometimes they don’t even live in a module!).
I’d encourage everyone to add randomization to their test runs, especially on large projects where the tests might have some dark corners. If you have any suggestions for pytest-randomly, its GitHub issues are always open.
© 2019 All rights reserved.