Presentation: How Machines Help Humans Root Cause Issues @Netflix
Share this on:
What You’ll Learn
-
Learn tips and techniques on building success tools for developers.
-
Understand how Netflix uses statistics and machine learning to enhance some of the tooling.
-
Learn approaches to pairing automation with human feedback to reduce mean time to detect and resolve issues more quickly.
Abstract
Automated root cause detection represents a holy grail goal for many systems monitoring tools. It’s also an extremely challenging domain. A successful approach must generalize well from limited examples, handle highly dimensional data, understand the application domain, and perform well in a real time environment where baseline behavior changes over time. Those are all problems humans are good at, but state of the art machine learning approaches often struggle with.
In this talk, we’ll discuss ways to build tools designed to enhance the cognitive ability of humans through automated analysis to speed root cause detection in distributed systems. We’ll focus on examples from large scale systems at Netflix. In particular, we’ll focus on the systems directly involved in browsing and playing Netflix movies, and how pairing automation with human feedback reduces mean time to detect and resolve issues.
You're at Netflix. What team are you working on, and what's the focus of the work that you do?
I work on a team called operational insights.
Netflix basically has a structure where there are the teams that are directly in the line of fire. These are teams that build systems where you can't play a movie unless their servers are working correctly.
Then there are infrastructure teams that build a whole bunch of stuff that supports everyone else. The infrastructure team's servers are very mission critical, so there is a need for a sort of an application specific bridge. Where basically a team is not directly in the line of fire, so we have the opportunity to build better tools.
An example of why this is useful is maybe someone (who is in the line of fire) writes a quick script that solves an immediate problem they're having but they probably don't have the bandwidth to take that script (if it's useful) and turn it into a broader tool that's more reusable. So that's kind of where we come in. We swoop in and turn these sort of nuggets of ideas into usable broadly accessible tools for all the teams that are directly in the line of fire.
Your talk is about helping with root cause using operational data. Is this a machine learning team?
What's interesting about our team is we didn't initially start with machine learning. Initially, we were a team that built straightforward tools for analyzing problems, making data lookup tools, tools for dashboards, and that sort of thing. Initially, that was successful, but very quickly it became unwieldy for people to do this without help from statistics and machine learning.
There's a couple of reasons for that. Netflix is built a bunch of microservices. It's a fairly large company, so if there are three teams involved in an interaction, just getting to the point where you can figure out which team is responsible for an interaction failing takes a decent amount of work. There's just a lot you have to look at to figure that out. So having machine learning and analytics to help surface those things became essential. That's sort of how we worked into it.
One takeaway from our experience that might be interesting to people in the room is you know you don't have to start over and AI/ML tooling from scratch. What you can do is take your existing tools and then apply on top of them machine learning. So build ML into existing tools. That's probably a better approach to getting adoption and success.
In your abstract you say this talk discusses ways to build tools designed to enhance cognitive ability through automated analysis. Can you give me an example of what you might discuss?
One of them has to do with providing context when troubleshooting an individual trace. Let's say you are a device manufacturer, and you're trying to certify your Netflix device. What if something goes wrong in that trace? It's in the middle of a sea of traffic if this is on production right. So how can we actually get a full sense of everything that the person did and why it might have gone wrong?
One thing people talk a lot about is capturing trace data. Zipkin as a popular tool. Dapper was a very popular paper a few years back. Those tools are useful. But what we found is actually the most important thing is the context about the trace. The presentation of the data in a way that's digestible. If you just think about it even playing one movie, it's a stateful interaction that lasts half an hour. So if it's just a list of traces, it would still overwhelm you. Being able to basically make sense of your data and build a tool around that I think is the first step.
What do you feel is the most important trend in software today?
I think it's that the golden age of machine learning. I think that the advances in Deep Learning will eventually be important even in this space. The reason Deep Learning doesn't work for us right now is that it requires large datasets. There's a lot of work in the last few years that's very promising about one shot deep learning (inferring rules or building reasoning maps) and that sort of thing. I think once that stuff is sort of at a more proven stage this whole space will go through a radical revolution.
Similar Talks
Tracks
-
Microservices: Patterns & Practices
Evolving, observing, persisting, and building modern microservices
-
Developer Experience: Level up Your Engineering Effectiveness
Improving the end to end developer experience - design, dev, test, deploy, operate/understand. Tools, techniques, and trends.
-
Modern Java Reloaded
Modern, Modular, fast, and effective Java. Pushing the boundaries of JDK 9 and beyond.
-
Modern User Interfaces: Screens and Beyond
Zero UI, voice, mobile: Interfaces pushing the boundary of what we consider to be the interface
-
Practical Machine Learning
Applied machine learning lessons for SWEs, including tech around TensorFlow, TPUs, Keras, Caffe, & more
-
Ethics in Computing
Inclusive technology, Ethics and politics of technology. Considering bias. Societal relationship with tech. Also the privacy problems we have today (e.g., GDPR, right to be forgotten)
-
Architectures You've Always Wondered About
Next-gen architectures from the most admired companies in software, such as Netflix, Google, Facebook, Twitter, Goldman Sachs
-
Modern CS in the Real World
Thoughts pushing software forward, including consensus, CRDT's, formal methods, & probalistic programming
-
Container and Orchestration Platforms in Action
Runtime containers, libraries, and services that power microservices
-
Finding the Serverless Sweetspot
Stories about the pains and gains from migrating to Serverless.
-
Chaos, Complexity, and Resilience
Lessons building resilient systems and the war stories that drove their adoption
-
Real World Security
Practical lessons building, maintaining, and deploying secure systems
-
Blockchain Enabled
Exploring Smart contracts, oracles, sidechains, and what can/cannot be done with blockchain today.
-
21st Century Languages
Lessons learned from languages like Rust, Go-lang, Swift, Kotlin, and more.
-
Empowered Teams
Safely running inclusive teams that are autonomous and self-correcting