Presentation: Peloton - Uber's Webscale Unified Scheduler on Mesos & Kubernetes

Track: Data Engineering for the Bold

Location: Majestic Complex, 6th fl.

Duration: 5:25pm - 6:15pm

Day of week: Tuesday

Share this on:

Abstract

With the increasing scale of Uber’s business, efficient use of cluster resources is important to reduce the cost per trip. As we have learned when operating Mesos clusters in production, it is a challenge to overcommit resources for latency-sensitive services due to their large spread of resource usage patterns. Uber also has significant demand on running large-scale batch jobs for marketplace intelligence, fraud detection, maps, self-driving vehicles etc.  

In this talk, we will present Peloton, a Unified Resource Scheduler for collocating heterogeneous workloads in shared Mesos clusters. The goal of Peloton is to manage compute resources more efficiently while providing hierarchical max-min fairness guarantees for different teams. Peloton schedules large-scale batch jobs with millions of tasks and also supports distributed TensorFlow jobs with thousands of GPUs.

Speaker: Mayank Bansal

Staff Engineer @Uber

Mayank Bansal is currently working as a Staff Engineer at Uber in data infrastructure team. He is co-author of Peloton. He is Apache Hadoop Committer and Oozie PMC and Committer. Previously he was working at Ebay in hadoop platform team leading YARN and MapReduce effort. Prior to that he was working at Yahoo and worked on Oozie.

Find Mayank Bansal at

Speaker: Min Cai

Sr Staff Engineer @Uber

Min Cai is a Sr Staff Engineer in Compute Platform team at Uber working on all-active data centers, cluster management and micro-service deployment systems. He received his Ph.D. degree in Computer Science from Univ. of Southern California. Before joining Uber, he was a Sr. Staff Engineer at VMware working on live VM migration (vMotion) and virtual infrastructure management (vCenter). He has published over 20 journal and conference papers, and holds 6 US patents.

Find Min Cai at

Similar Talks

Psychologically Safe Process Evolution in a Flat Structure

Director of Software Development @Hunter_Ind

Chris Lucian

Let's talk locks!

Software Engineer @Samsara

Kavya Joshi

Graceful Degradation as a Feature

Director of Product @GremlinInc

Lorne Kligerman

Scaling Infrastructure Engineering at Slack

Senior Director of Infrastructure Engineering @Slack

Julia Grace

Liberating Structures at Capital One

Agile Coach, Engineering @CapitalOne

Greg Myers

Making 'npm install' Safe

Software Engineer @agoric

Kate Sills

Rocket-fueled Engineering & Technology Transformation @WeWork

Fellow Engineer, Developer Platform @WeWork

Hugo Haas

Tracks

Monday, 24 June

Tuesday, 25 June

Wednesday, 26 June