pytest-randomly history
My plugin pytest-randomly was recently moved into the pytest-dev organization on GitHub, making it a bit “more official” as a pytest plugin. Thanks to Bruno Oliveira for suggesting it, Florian Bruhin and Bruno for approving it on the pytest-dev mailing list, and Gordon Wrigley for helping with its development.
In celebration I thought I’d explain a bit more of the background behind it. pytest-randomly really combines two functions:
- Controlling the random seed between test runs, which is useful when using a tool like Factory Boy to generate test data. By allowing the same seed to be used again, failures can be debugged. See more in my blog post on it.
- Reordering tests randomly, to discourage order-dependency, which can be common with certain fixture patterns touching global state like a database
For YPlan, we needed random seed control first. We added Factory Boy to shrink the test code needed to set up Django model instances, and to get more value from the tests by covering a wider range of cases between runs. We were using nose at the time, and implemented a plugin to reset the seed at the start of each test and a flag to control in just a few lines of code.
Later on we added the test reordering code, due to problems with Django’s setUpTestData. This hook allows you to perform setup once at the start of all the tests in a test class, and keep it between the individual test functions with rollback at the database layer. The benefits can be huge as you can perform the same setup for a bunch of tests once instead of tens of times. The problem we encountered was mismatch between python and the database. Consider these tests:
class MyTests(TestCase):
@classmethod
def setUpTestData(cls):
super().setUpTestData()
# DB rolled back to this state between tests
cls.book = Book.objects.create(name="Verily, A New Hope")
def test_1(self):
self.book.name = "The Empire Striketh Back"
self.book.save()
do_some_assertions()
def test_2(self):
do_some_other_assertions()
The problem is that under the written test ordering, when test_2 runs its self.book object will be inconsistent between python and the database. The database has been rolled back to the state from setUpTestData, but on the MyTests object, self.book.name will still be 'The Empire Striketh Back' - there is no process to undo the changes from test_1 in Python. When this causes test failures it’s a bad case of action-at-a-distance, with state leaking between tests.
This can be avoided by never caching model instances on the test cases, or always copy.deepcopy()ing them before modifying them inside a test function, but both are easy to forget. And there is always other global state that can be modified and not reset, like the contents of any module (yay Python 😁).
Order randomization is a way of protecting against this problem. If tests leak state and pass on one run, it’s unlikely they’ll pass on the next run as the order will be re-randomized.
We implemented randomization of order on nose by taking some existing code from ‘Massive Wombat’ (?) on Google Code and Nick Loadholtes on GitHub, merging it with our random seed reset plugin, and tidying it up to work with all the kinds of tests we had. After a while I had the chance to open source it as nose-randomly. Naturally, when we moved to pytest, this was rebuilt as pytest-randomly 👌
The main development on pytest-randomly since has been to make it reset the random generators used by some popular libraries, and to make it work with all the kinds of test items that pytest supports (sometimes they don’t even live in a module!).
I’d encourage everyone to add randomization to their test runs, especially on large projects where the tests might have some dark corners. If you have any suggestions for pytest-randomly, its GitHub issues are always open.
Learn how to make your tests run quickly in my book Speed Up Your Django Tests.
One summary email a week, no spam, I pinky promise.