For any company to run on the cloud they need assurances that their workloads, services, and data will be always available and secure. To be able to provide such guarantees, application developers and cloud providers need to perform extensive verification across a number of distributed services. Traditional testing tools were not designed to verify the resiliency of such systems.

At Microsoft, we actively develop and use fault injection to test and break our services. By doing this we identify failure points, design better detection, and build mitigations which allow us to auto-heal when real issues arise. Fault injection can span the whole stack: from applications to hardware and the network, from VMs to datacenters.

Developing a fault injection system is tricky, but utilizing it effectively is a magnitude of order harder. It’s important for engineers to embrace the fault injection culture and be trained to leverage it in all phases of development and maintenance. Only then can we significantly reduce the TTD/TTM (time-to-detect/mitigate). We will present our learnings for designing and using fault injection systems to maintain highly-available cloud-scale applications and the cultural change necessary to enable them.