You are viewing content from a past/completed QCon

Presentation: Peloton - Uber's Webscale Unified Scheduler on Mesos & Kubernetes

Track: Data Engineering for the Bold

Location: Soho Complex, 7th fl.

Duration: 4:10pm - 5:00pm

Day of week: Tuesday

Share this on:

This presentation is now available to view on InfoQ.com

Watch video with transcript

Abstract

With the increasing scale of Uber’s business, efficient use of cluster resources is important to reduce the cost per trip. As we have learned when operating Mesos clusters in production, it is a challenge to overcommit resources for latency-sensitive services due to their large spread of resource usage patterns. Uber also has significant demand on running large-scale batch jobs for marketplace intelligence, fraud detection, maps, self-driving vehicles etc.  

In this talk, we will present Peloton, a Unified Resource Scheduler for collocating heterogeneous workloads in shared Mesos clusters. The goal of Peloton is to manage compute resources more efficiently while providing hierarchical max-min fairness guarantees for different teams. Peloton schedules large-scale batch jobs with millions of tasks and also supports distributed TensorFlow jobs with thousands of GPUs.

Speaker: Mayank Bansal

Staff Engineer @Uber

Find Mayank Bansal at

Speaker: Apoorva Jindal

Senior Software Engineer @Uber

Find Apoorva Jindal at

Proposed Tracks

  • Trouble-Shooting in Production

  • Disrupting Technology on Wall Street

  • Resilience vs Failure in Architecture

  • The Weeds of Distributed File Systems

  • Organizational Agility

  • Product & Customer Focused Teams

  • Just Culture (Blameless Culture)

  • Modern CS in the Real World

  • Architectures You’ve Always Wondered About

  • Machine Learning and AI in the New Decade

  • Evolving Java - Including K8s/Containers, Kotlin and Impact on AOT

  • Ethical Considerations in Software

  • Microservices and Scalability

  • Container Slinging

  • Native Compilation Is Back (A Look at Non-Vm Compilation Targets)