Presentation: Canopy: Scalable Distributed Tracing & Analysis @Facebook
Share this on:
Abstract
How do you understand the performance of a request that is executed in a large-scale system, potentially fanning out across thousands of machines and services? To answer this question at Facebook, we built a distributed tracing framework, Canopy, which has provided visibility into an otherwise intractable problem.
In this talk we present Canopy, Facebook’s performance and efficiency tracing infrastructure. Canopy recoards causally related events across the end-to-end execution path of requests, including from browsers, mobile applications, and backend services. Canopy processes traces in near real-time, derives user-specified features, and outputs to datasets that aggregate across billions of requests. At Facebook, Canopy is used to query and analyze performance and efficiency data in real-time.
Canopy addresses three challenges we have encountered: (1) supporting the range of execution and performance models used by different components of the Facebook stack; (2) supporting interactive ad-hoc and real-time analysis of trace data; and (3) operating at massive scale - Canopy currently records and processes over 1 billion traces per day.
We conclude by discussing lessons learned applying Canopy to a wide range of use cases at Facebook and present case studies of its use in solving various performance and efficiency challenges
Tracks
-
Blameless Culture
Absorb the lessons learned from failures and outages in a human-centric process.
-
Modern CS in the Real World
Learn how companies are applying recent CS research to tackle concurrency, distributed data, and coordination.
-
Architectures You’ve Always Wondered About
Next-gen architectures from the most admired companies in software.
-
Bare Knuckle Performance
Learn from practitioners on the challenges and benefits of architecting for performance and much more.
-
Java - The Interesting Bits
Learn the new features in the recent and near-future releases of Java and the JVM and what they offer.
-
Ethical Considerations in Consciously Designed Software
Design considerations for various contexts, locations, security and privacy requirements.
-
Operating Microservices
Learn from practitioners operating and evolving systems in performance demanding environments.
-
Security
Learn how to make security an inherent part of the software development process.
-
Native Compilation Is Back (A Look at Non-Vm Compilation Targets)
Issues with native compilation for in browser-based and server-side environments
-
Trouble-Shooting in Production
Learn debugging strategies for complex and high stakes environments where standard debuggers and profilers fail.
-
Predictive Architectures and ML
Explore the systems and designs covering the full loop from machine learning to inferencing.
-
Data Engineering on the Bleeding Edge
Explore the latest trends in data engineering that help improve the life of the developer
-
Production Readiness
Observability, emergency response, capacity planning, release processes, and SLOs for availability and latency.
-
Humane Leadership
A look at leadership with an emphasis on empathy, taking chances and building other leaders within organizations and teams
-
Devex & Teams
Explore how to reduce developer friction between teams and stakeholders.