Failure is a fact of life, so we design our system to be fault-tolerant at all levels. In practice, however, some components almost never fail. As the product grows, these components are increasingly stressed in new and different ways; when they ultimately do fail they create outages for which we are unprepared. We thought we were designing for failure, but the design didn't include failures at this level. At Google, some of our most exciting production snafus involve large and unpredictable...

Distributed or "cloud" computing involves many moving parts, any of which can break or fail. Succeeding in this environment requires embracing failure, not running or hiding from it. To do this requires challenging our instincts with radical ideas. Tom will highlight some of the most radical advice from the new book “The Practice of Cloud System Administration”.

Topics will include: create resiliency at the most economic level, do risky procedures often, and create a blameless culture...