Comcast’s TV products serve tens of millions of customers and are powered by a suite of dozens of services that are continuously developed and operated by hundreds of technical staff. While we have enjoyed many of the touted benefits of a microservice architecture--looser coupling between teams, independent deployments--we have also encountered the corresponding reliability challenges. Delivering business value in this environment can seem like hacking your way through the wilderness at times.

In this talk, we’ll go over some strategies for surviving in a jungle of partial failures. Each survival tip will be explained through a concrete example, or "adventure story", from our experience. By the end of the talk, attendees will be familiar with the pitfalls of partial failures—the main technical weakness of a microservice architecture—but will also be armed with techniques to successfully avoid them.