Adam Johnson

Home | Blog | Projects | Colophon

Running CloudFormation Drift Detection on All Your Stacks

2019-06-05

Fighting drift in the cloud

CloudFormation’s stack drift detection feature is useful. It discovers ways your infrastructure that you beautifully set up with Infrastructure-as-Code has been fiddled with manually. Often this results from a “quick temporary fix” being applied manually on the web console at 2am, then forgotten about.

By running drift detection on a stack you get a report of things that have been changed, and then can go tell off the process-obviating miscreant. (Unless the miscreant is you, in which case you can just feel embarrassment. We’ve all been there.)

Currently it supports a limited selection of resource types but even so they are useful to check.

How to Drift

In typical AWS fashion, the feature exists but it’s up to us to integrate it ourselves. We can click manually on each stack and hit “Detect Drift” to run it:

Web console

Then once it’s done, you have to click through each stack to see the results again. This gets old as soon as you have more than one stack.

In an ideal world, I’d have it run automatically on every ready stack daily and get alerts to any new drift. For example, this could be done with a Lambda function and CloudWatch Events.

But for now, let’s look at a way of running drift detection on every stack without getting RSI from all that clicking.

Fire up that AWS CLI

Update (2019-07-12) As pointed out by jeshan on reddit, it's also possible to automate this with AWS Config with the cloudformation-stack-drift-detection-check rule.

First, trigger drift detection on each stack in the current region with this shell one-liner:

for stack in $(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE --query 'StackSummaries[].StackName' --output text); do aws cloudformation detect-stack-drift --stack-name "$stack"; done

To unpack it:

I’ve tested this on ZSH and Bash with two AWS accounts, but this might break with non-alphanumeric stack names, some shells, or some shell configurations.

When you run it, you’ll see output with a result from each call to detect-stack-drift, for example:

{
    "StackDriftDetectionId": "00000000-0000-0000-0000-000000000000"
}
{
    "StackDriftDetectionId": "00000000-0000-0000-0000-000000000001"
}

Second, twiddle your thumbs a bit. Maybe make a cup of tea, or coffee. Drift detection takes a little while, and may be internally limited if you’re doing it on a lot of stacks.

Third and finally, let’s list the stack names and results in a table with the CLI. This is much more convenient than the web console, which doesn’t show the drift status on the list view, so you have to click through each damn one. Also we can sort them by name, so much easier!

Run this single command:

aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE --query 'sort_by(StackSummaries, &StackName)[*].[StackName, StackStatus, DriftInformation.StackDriftStatus, DriftInformation.LastCheckTimestamp]' --output table

This will give you output that looks like this:

----------------------------------------------------------------------------------
|                                   ListStacks                                   |
+---------------------+------------------+----------+----------------------------+
|  example-com        |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:26.528Z  |
|  example-org        |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:22.004Z  |
|  do-not-delete      |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:23.157Z  |
|  illuminati-funding |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:13.454Z  |
+---------------------+------------------+----------+----------------------------+

If some of the stacks haven’t completed drift detection yet as per the third and fourth columns, you can just rerun this listing until they have.

False positives

I’ve found a couple cases with my current modestly-sized client where false positive stack drift is detected:

It seems there’s a little way to go on the feature. That said, false positives are better than false negatives here.

If you want to build alerting around drift, you might want to add in a way to ignore false positives.

Fin

Enjoy stopping drift,

—Adam


Subscribe via RSS, Twitter, or email:

Related posts:

Tags: aws, cloudformation, commandline