Testing Boto3 with Pytest Fixtures2019-04-22
This is a recipe I’ve used on a number of projects. It combines Pytest fixtures with Botocore’s Stubber for an easy testing experience of code using Boto3. (Botocore is the library behind Boto3.)
Imagine we have a Boto3 resource defined in
And a function to test in
We want to write tests for
In our tests we want to ensure that:
- The real S3 service is never touched
- We can make accurate assertions on all S3 requests
- We expect all the requests the system makes
We could turn to
unittest.mock, but its mocking is heavy-handed and would remove
boto3’s argument checking.
(Boto3 is not
Instead let’s use Botocore’s Stubber.
A Stubber can temporarily patch a client to avoid contacting its service and instead interact with a queue of pre-declared responses.
We can set one up in a Pytest fixture in a file called
tests/conftest.py like so:
- We put the fixture in the
conftest.pyin the base
testsdirectory so it is available in all tests.
- We set
autouse=Trueso that Pytest applies the fixture to every test, regardless of whether the test requests it.
- We access the
boto3Resource’s underlying Client with
.meta.client. If our application used a Client we could stub it client directly.
- We yield the
stubberas the fixture object so tests can make use of it.
- We use the stubber’s
assert_no_pending_responses()at the end to check that the test made every expected request.
We could use the fixture in a test file called
tests/s3_functions/test_do_something.py like so:
This follows the “Arrange, Act, Assert” (AAA) pattern for writing tests.
To arrange, we declare the expected botocore responses and their parameters.
Here there is just one, a call to
When given as positional arguments, the
service_response arguments are reversed, but I prefer to use keyword arguments and put them in natural “request, response” order.
expected_params is also optional, but it improves our test coverage to include it.
If it’s hard to compute one or more of the expected parameters, you can use
botocore.stub.ANY as a stand-in value which always compares equal.
To act, we call the tested function with test parameters. The Stubber will match incoming requests against the declared requests and return the matching responses. We should declare the requests in expected order, since they will act as a queue.
To assert, we make some standard Pytest assertions.
I didn’t fill them in here.
During teardown the fixture runs
stubber.assert_no_pending_responses() as a final assertion.
Testing AWS Errors
We can also emulate the AWS service raising an error by using the Stubber’s
We could use it to test
do_something’s behaviour with a missing S3 key like so:
Finding the value for
service_error_code value is a matter of looking in the relevant service’s API documentation, for example the S3 Error Responses page.
Or, more robustly, cause that error manually with Boto3 and find it in the raised exception.
pytest.raises to capture the
This prevents the exception failing the test and allows us to make assertions about it.
I hope this recipe helps you build more robust AWS integrations,
Working on a Django project? Check out my book Speed Up Your Django Tests which covers loads of best practices so you can write faster, more accurate tests.
One summary email a week, no spam, I pinky promise.
Tags: aws, pytest, python
© 2021 All rights reserved.