Running CloudFormation Drift Detection on All Your Stacks2019-06-05
CloudFormation’s stack drift detection feature is useful. It discovers ways your infrastructure that you beautifully set up with Infrastructure-as-Code has been fiddled with manually. Often this results from a “quick temporary fix” being applied manually on the web console at 2am, then forgotten about.
By running drift detection on a stack you get a report of things that have been changed, and then can go tell off the process-obviating miscreant. (Unless the miscreant is you, in which case you can just feel embarrassment. We’ve all been there.)
Currently it supports a limited selection of resource types but even so they are useful to check.
How to Drift
In typical AWS fashion, the feature exists but it’s up to us to integrate it ourselves. We can click manually on each stack and hit “Detect Drift” to run it:
Then once it’s done, you have to click through each stack to see the results again. This gets old as soon as you have more than one stack.
In an ideal world, I’d have it run automatically on every ready stack daily and get alerts to any new drift. For example, this could be done with a Lambda function and CloudWatch Events.
But for now, let’s look at a way of running drift detection on every stack without getting RSI from all that clicking.
Fire up that AWS CLI
First, trigger drift detection on each stack in the current region with this shell one-liner:
To unpack it:
- We start with a shell
forloop. This uses the keywords
- The loop goes over the output of the
aws cloudformation list-stackscommand. We filter them with
--stack-status-filterto only the stacks in “green” states (yes we have to list all these states explicitly…). We then reduce the output to a single line of space separated stack names using
--query, which filters the data with JMESPath, and
--output text, which uses plain text rather than JSON output.
- The shell’s for loop splits the
list-stacksoutput by whitespace and runs the loop body for each item set as the
- We run
aws cloudformation detect-stack-driftin the loop body, templated with the
I’ve tested this on ZSH and Bash with two AWS accounts, but this might break with non-alphanumeric stack names, some shells, or some shell configurations.
When you run it, you’ll see output with a result from each call to
detect-stack-drift, for example:
Second, twiddle your thumbs a bit. Maybe make a cup of tea, or coffee. Drift detection takes a little while, and may be internally limited if you’re doing it on a lot of stacks.
Third and finally, let’s list the stack names and results in a table with the CLI. This is much more convenient than the web console, which doesn’t show the drift status on the list view, so you have to click through each damn one. Also we can sort them by name, so much easier!
Run this single command:
This will give you output that looks like this:
If some of the stacks haven’t completed drift detection yet as per the third and fourth columns, you can just rerun this listing until they have.
I’ve found a couple cases with my current modestly-sized client where false positive stack drift is detected:
- Tags added automatically by AWS being reported as drift, e.g. when using a launch template, EC2 adds the tags
- Load balancer listener rules not pulling every detail back in ‘Actual’ e.g. a redirect rule being picked up by drift detection, but not its
It seems there’s a little way to go on the feature. That said, false positives are better than false negatives here.
If you want to build alerting around drift, you might want to add in a way to ignore false positives.
Enjoy stopping drift,
© 2019 All rights reserved.