Slack

What Breaks Our Systems: A Taxonomy of Black Swans

Tell me a bit about yourself.

I'm a Senior Staff Production Engineer at Slack, I've just started here. In terms of the work that I do here I don't know yet. This is my second week. At Google, I used to run large production services. I cared about primarily reliability and availability: is my service up, is my service performing well, are my users happy?

Tell me why you wrote this talk.

One of the things that I think that we are not great at as an industry is thinking about outlier events, particularly with the way that SRE has gained popularity in the last three or four years. People started talking a lot about SLOs, error budgets, and maintaining those on a week to week basis or month to month basis. I think this is great but I think that one of the things that gets lost in that view of the world is thinking about what are my systemic risks? What are the things that can go really, really wrong with my systems. What are the things that could take me down for hours, days, maybe even longer, potentially even destroy the business. I think that worrying about those kinds of scenarios should be a core part of the job of site reliability engineers, production engineers, senior engineers. That's what this talk is about.