Presentation: Heretical Resilience: To Repair is Human

I'm currently working at Travis CI where I'm the lead of the build environment team. This team is working on the environment that allows our customers to run their builds - making sure that we can create, test, and update the environments where customer builds get run in a reliable manner, so that customers can continue to test as new software releases are coming out. I've been at Travis for just under a year now, and before that I spent three years doing web operations at Etsy, where I focused on provisioning, monitoring, and configuration management.

One thing that I've thought about a lot in terms of monitoring and on calls specifically is the idea of alert design. What information are we putting into the alerts? Particularly putting some context in there - let's look at a typical example of a disk space alert. If you get a disk space alert and it says the disk is 90% full and it’s 3am, how urgent is that alert actually? If it’s been creeping up slowly to 90%, it can likely wait until morning. Or did it just spike up and if if you don't do something about it now, something is going to go terribly wrong? Adding context means something like being able to put a graph into that alert, because just number going over a threshold doesn't necessarily tell you what you need to know.

A lot of what I'm talking about is how we approach automation and orchestration. The Apache SNAFU is a story about when automation - in this case, configuration management - went terribly wrong, and how people were responding to that. When you automate away all the tedious bits or the boring bits of something, the parts that are left over tend to be more complex. In addition to that, over time things may have been automated for so long that people forget what is actually going on under the hood. When that goes wrong, when these complex tools that we've created to manage these complex systems go wrong, as every computer thing inevitably does at some point or another, how do we respond to that? Do we know how to repair our automation as well as the systems being automated? In addition, I want people to think about how we can make sure that we are also learning from these incidents. I've worked at places over the years where everyone was always so busy firefighting that nobody ever had the time to do anything more than that, more than just putting a quick bandaid fix on something. When you're in the midst of firefighting, sometimes that's all you have time to do in the moment. But that's not good in the long run- it’s not how you develop organizational learning, it's not how you develop resilience. Instead it usually ends up making things more and more fragile.