THOMISSA: I am a technical project manager in reliability on the infrastructure team.

THOMISSA: We are focused on DRTs (Disaster Recovery Tests). We intentionally break things to make them better. This is something that we are working on as a company, and we are constantly looking to improve. It’s an ongoing cycle of having things break, fixing them, testing the fixes, realizing we need to break other things to test the same thing, and then working on those things which generally breaks other things. Improvement from failure.

THOMISSA: One of the things that we see is that there is a lot of inter-system dependencies.

TAMMY: We look at the edge (the edges between systems rather than looking at the system just as themselves). Also, the biggest difference at Dropbox (compared to anywhere else that I have worked) is that teams are empowered to run their own DRTs on their own schedule. We are encouraged to run them as frequently as we can, measure and put forward ideas like 'this is what we are going to run and this is why'. think this approach is really important. It is a different way to look at it because the teams are empowered to run DRTs. Then the reliability team helps by giving this methodology to help navigate who is running jobs at what time so you don’t have clashes. That is really important too. If everyone runs on the same day and don’t know about each other is doing, it could be very dangerous.

THOMISSA: Well, from a cultural standpoint: how can anyone on a team, SRE or otherwise, think about engineering for failure and why should they? How can these team members find their weaknesses and engage with other teams to make a more robust product overall? We hope to frame a way to build tests, break them into multiple pieces, prioritize them, and execute.