Presentation: A Series of Unfortunate Container Events @Netflix

Track: Immutable Infrastructures: Orchestration, Serverless, and More

Location: Broadway Ballroom South Center, 6th fl.

Duration: 10:35am - 11:25am

Day of week: Wednesday

Level: Intermediate

Persona: Architect, DevOps Engineer


Project Titus is Netflix's container runtime on top of Amazon EC2. Titus powers algorithm research through massively parallel model training, media encoding, data research notebooks, ad hoc reporting, NodeJS UI services, stream processing and general micro-services. As an update from last year's talk, we will focus on the lessons learned operating one of the largest container runtimes on a public cloud. We'll cover the migration we've seen of applications and frameworks from VM's to containers. We will cover the operational issues with containers that only showed after we reached the large scale (1000's of container hosts, 100's of thousands of containers launched weekly) we are currently supporting. We'll touch base on the unique features we have added to help both batch and microservices run across a variety of runtimes (Java, R, NodeJS, Python, etc) and how higher level frameworks have taken avantage of Titus's scheduling capabilities.


QCon: What’s the motivation for your talk?

Andrew & Amit: Last year we talked about why we choose to build Titus, our container management platform. We also talked about the architecture and implementation of Titus. This year, we wanted to talk about lessons learned running and continuing to evolve this critical system at Netflix. We wanted to share the lessons learned providing Netflix required availability running this system at our unique levels of scale.

QCon: What’s the level & core persona?

Andrew & Amit: We expect most attendees who are considering containers as part of their infrastructure to benefit with the lessons learned across the varied use cases we've seen benefit from containers. Additionally, those growing container environments from test and development will learn key lessons of what it takes to run containers in production. Finally, lessons learned only at Netflix scale will be presented. These final lessons of worldwide scale are usually academically interesting to most engineers.

QCon: What 3 actionable things do you want persona to walk away with?

Andrew & Amit:

1. What aspects to consider after deploying an off-the-shelf container management platform

2. How to think about reliability in the context of a large scale distributed container management platform

3. What levels of scale are possible, not through synthetic benchmarks, but real world container deployments

QCon: Ask and answer an interesting question of your choice that you think an attendee might have after reading your abstract?

Andrew & Amit:

Q: Why is Netflix's container management platform different from other open source container management platforms?

A: Given our existing VM based cloud native infrastructure, we approached container management as an addition to our existing cloud platform -- instead of a replacement. This means a key to the success of Titus was deciding what Titus would not do, leveraging the full value other infrastructure teams at Netflix provide. This also meant leveraging Amazon Web Services (AWS) deeply seamlessly integrating VM's and containers while supporting existing operations and security models in AWS. Finally, it was important to consider how we choose workloads that benefited from containers as compared to pushing all workloads to containers.

Speaker: Andrew Spyker

Open Source Coordinator @Netflix

Andrew has worked on the Cloud Platform team as Netflix for the past two years. He joined Netflix to help with performance and scalability across the core building blocks of Netflix’s cloud platform. Since joining Netflix, he has helped with not only performance but also architecture, open source, and container strategy. More recently Andrew has been an engineer as well as product manager for the container cloud initiative (Project Titus).

Find Andrew Spyker at

Speaker: Amit Joshi

Senior Software Engineer @Netflix

Find Amit Joshi at


Monday, 26 June

Tuesday, 27 June

Wednesday, 28 June