You are viewing content from a past/completed QCon

Presentation: Canopy: Scalable Distributed Tracing & Analysis @Facebook

Track: Architectures You've Always Wondered About

Location: Broadway Ballroom North, 6th fl.

Duration: 11:50am - 12:40pm

Day of week: Wednesday

Level: Intermediate

Persona: Architect, Developer

Share this on:

Abstract

How do you understand the performance of a request that is executed in a large-scale system, potentially fanning out across thousands of machines and services? To answer this question at Facebook, we built a distributed tracing framework, Canopy, which has provided visibility into an otherwise intractable problem. 

In this talk we present Canopy, Facebook’s performance and efficiency tracing infrastructure. Canopy recoards causally related events across the end-to-end execution path of requests, including from browsers, mobile applications, and backend services. Canopy processes traces in near real-time, derives user-specified features, and outputs to datasets that aggregate across billions of requests. At Facebook, Canopy is used to query and analyze performance and efficiency data in real-time.   

Canopy addresses three challenges we have encountered: (1) supporting the range of execution and performance models used by different components of the Facebook stack; (2) supporting interactive ad-hoc and real-time analysis of trace data; and (3) operating at massive scale - Canopy currently records and processes over 1 billion traces per day.   

We conclude by discussing lessons learned applying Canopy to a wide range of use cases at Facebook and present case studies of its use in solving various performance and efficiency challenges

Speaker: Haozhe Gao

Software Engineer @Facebook

Edison is a Software Engineer at Facebook on the Canopy team, Facebook's distributed tracing framework. Prior to joining Facebook, he interned at Palantir. Edison graduated from Carnegie Mellon University with a BS in Mathematics and a BS in Computer Science.

Find Haozhe Gao at

Speaker: Joe O’Neill

Software Engineer @Facebook

Joe is a Software Engineer at Facebook on the Canopy team, Facebook's distributed tracing framework. Prior to joining Facebook, he interned at Microsoft. Joe graduated from Northeastern University with a BS in Computer Science and Business Administration.

Find Joe O’Neill at

Tracks

  • Blameless Culture

    Absorb the lessons learned from failures and outages in a human-centric process.

  • Modern CS in the Real World

    Learn how companies are applying recent CS research to tackle concurrency, distributed data, and coordination.

  • Architectures You’ve Always Wondered About

    Next-gen architectures from the most admired companies in software.

  • Bare Knuckle Performance

    Learn from practitioners on the challenges and benefits of architecting for performance and much more.

  • Java - The Interesting Bits

    Learn the new features in the recent and near-future releases of Java and the JVM and what they offer.

  • Ethical Considerations in Consciously Designed Software

    Design considerations for various contexts, locations, security and privacy requirements.

  • Operating Microservices

    Learn from practitioners operating and evolving systems in performance demanding environments.

  • Security

    Learn how to make security an inherent part of the software development process.

  • Native Compilation Is Back (A Look at Non-Vm Compilation Targets)

    Issues with native compilation for in browser-based and server-side environments

  • Trouble-Shooting in Production

    Learn debugging strategies for complex and high stakes environments where standard debuggers and profilers fail.

  • Predictive Architectures and ML

    Explore the systems and designs covering the full loop from machine learning to inferencing.

  • Data Engineering on the Bleeding Edge

    Explore the latest trends in data engineering that help improve the life of the developer

  • Production Readiness

    Observability, emergency response, capacity planning, release processes, and SLOs for availability and latency.

  • Humane Leadership

    A look at leadership with an emphasis on empathy, taking chances and building other leaders within organizations and teams

  • Devex & Teams

    Explore how to reduce developer friction between teams and stakeholders.