You are viewing content from a past/completed QCon

Presentation: Peloton - Uber's Webscale Unified Scheduler on Mesos & Kubernetes

Track: Data Engineering for the Bold

Location: Soho Complex, 7th fl.

Duration: 4:10pm - 5:00pm

Day of week: Tuesday

Share this on:

Abstract

With the increasing scale of Uber’s business, efficient use of cluster resources is important to reduce the cost per trip. As we have learned when operating Mesos clusters in production, it is a challenge to overcommit resources for latency-sensitive services due to their large spread of resource usage patterns. Uber also has significant demand on running large-scale batch jobs for marketplace intelligence, fraud detection, maps, self-driving vehicles etc.  

In this talk, we will present Peloton, a Unified Resource Scheduler for collocating heterogeneous workloads in shared Mesos clusters. The goal of Peloton is to manage compute resources more efficiently while providing hierarchical max-min fairness guarantees for different teams. Peloton schedules large-scale batch jobs with millions of tasks and also supports distributed TensorFlow jobs with thousands of GPUs.

Speaker: Mayank Bansal

Staff Engineer @Uber

Mayank Bansal is currently working as a Staff Engineer at Uber in data infrastructure team. He is co-author of Peloton. He is Apache Hadoop Committer and Oozie PMC and Committer. Previously he was working at Ebay in hadoop platform team leading YARN and MapReduce effort. Prior to that he was working at Yahoo and worked on Oozie.

Find Mayank Bansal at

Speaker: Apoorva Jindal

Senior Software Engineer @Uber

Apoorva Jindal is a Sr Software Engineer at Uber technologies working on micro-service deployment systems and cluster management. He received his PhD from the University of Southern California. He has published over 20 journal and conference papers and hold 2 US patents.

Find Apoorva Jindal at

Last Year's Tracks

Monday, 24 June

Tuesday, 25 June

Wednesday, 26 June