QCON NEW YORK 2017

Conference: Jun 26-28, 2017
Workshops: Jun 29-30, 2017

Presentation: Lessons Learned on Uber's Journey into Microservices

Track: 
Microservices and Monoliths

Location:

  • Salon D

Duration

Duration: 
2:55pm - 3:45pm

Day of week:

  • Wednesday

Level:

  • Intermediate

Persona:

  • Architect
  • CTO/CIO/Leadership
  • Developer
  • DevOps Engineer

More talks on:

Key Takeaways

  • Hear stories (good and bad) as a rapid growth company broke up a monolith into a series of Microservices.
  • Learn practices and lessons that will save you time and money if you get right before breaking up your monolith.
  • Understand the story around scalability and moving your architecture to Microservices.

Abstract

Uber had a monolith endearingly named "API" - which at the time comprised of the bulk of Uber's backend processing code. As this service was getting unwieldy to maintain - for scalability and a variety of reason - we launched on a journey to break this up into Microservices. Thus began "Project Darwin". Little did we realize what we were in for. Turns out that migrations are not easy, and take a long, long time. At Uber, the physical world depends on our service, and as such, we hold ourselves to a high standard for availability. That means when we build new systems, there are many phases of validating the new system before production traffic can be rerouted to it. With this talk, we hope to share our learning with you to help you when you undergo a large migration of your own!

Interview

Question: 
What is your role today?
Answer: 
I am a developer on Uber’s money team, which consists of collecting money from Uber riders after taking trips and then paying out the driver partners. I have been involved in both sides since joining Uber in January last year.
Question: 
Tell me a bit about the architecture your team uses.
Answer: 
Most of our code has been in a monolith up until this year, and we’ve broken it apart and spun up 15 to 20 services within the money team organization. Historically all of our code has been Python but the new microservices are starting to emerge in Go.
Question: 
What is the main topic of your talk?
Answer: 
This migration has been a huge undertaking and we’ve learned several things along the way. I am going to tell the war story of how this migration has gone, what we have learned, and how it could have gone more smoothly.
For example, the monolith held most of Uber’s backend code, but we also have many consumers that aren’t part of this monolith. Our migration has significantly impacted them as well since they have to migrate from calling what was once the monolith to a several new microservices. One thing that we have learned is to align early with your consumers. Let them know what is coming and how you are going to help them through the migration.
It took a lot of effort to convince our consumers later to undergo this migration, because for them, there is not much benefit. Many product teams are focused on improving their product -- they don't directly see much value added from changing one single API call into calls to 4, 5 or 10 microservices. Getting buy in from them early on will speed up this process a lot.
Also, a lot of difficulty came with not knowing who our consumers were. Building tooling up front would have helped consumer migration carry out more smoothly. We have built tooling once we realized it was necessary, but it would have helped if we had built these tools before we started the migration.
For example, a consumer may have previously made one API call that would extract data from many data models. Now, one of those data models has moved out into a microservice, but the consumer still needs all that data conglomerated into one response. Thus, we have now built a tool that splits the endpoint that used to touch all the data models into separate endpoints, with one of them talking to the new microservice. The tool then can take that microservice’s response, hit the other remaining endpoints that still are in the monolith, conglomerate the response and send it back to the consumer as if it hadn’t changed.
Question: 
What convinced Uber to replace the monolith with microservices?
Answer: 
It was an engineering wide decision and it was about scalability in three ways. Scaling our software with the growth of the company; the monolith was not going to scale with a single, writeable Postgres database. It was not going to scale with the growth of our business.
The second way of scalability was feature scaling. Adding new features to the old monolith was slow. It was hard to test and any mistake would have a huge impact, and the impact could be unexpected. If we rolled out something into the monolith and it unexpectedly affected other parts of it, then it could take down all of Uber’s back-end.
The third way is scaling the organization. This huge codebase wasn’t owned by anyone. Many teams had their code in there, but supporting this was really tough because there are just so many teams committing to it and the ownership model wasn’t really there. Also, if new developers joining Uber were tasked with doing anything to this monolith, it would be daunting and could potentially be catastrophic if a mistake was made.
Question: 
What do you mean the ownership model wasn’t there?
Answer: 
There were many intertwined dependencies in this monolith, and it was hard to define who owns which section of code. Some things were clear. If the money team added a feature in the monolith, then they would own that feature. But if there were many dependencies on many different data models in the monolith that the money team was using, and if that team wanted to change their data model, then who was responsible for migrating that? There was just no clear ownership on the monolith as a whole.
Question: 
What are the key takeaways of your talk?
Answer: 
One of the things that I want to touch on is the fact that monoliths can have many benefits. There was a reason that we put all of our code in this one monolith starting five or so years ago, but at a certain point stability is more important than development speed, which was one of the benefits of having this monolith. You have access to all types of data in one place. When Uber was a lot smaller, we could roll out features a lot more quickly than if we had to integrate with a whole bunch of different microservices.
But as this became unstable and unscalable, we needed to break it up for those three scalability reasons that I mentioned. Then, through this migration, we have learned several things about how to approach a migration of this size. I want to share with the audience what we have learned so that if they undergo a migration- either from monoliths to microservices or any migration- what that takes and what we have learned about how to make it go smoothly.

Speaker: Emily Reinhold

Software Engineer @Uber

Emily is a software engineer on Uber's Money team. Since joining Uber in early 2015, Emily has been involved in many aspects of money, including charging riders and paying driver partners. She has recently contributed to the effort to dismantle Uber's monolith while building its microservice architecture. Before Uber, she studied computer science at UC Berkeley.

Find Emily Reinhold at

Speaker page

Software Engineer at Uber

Similar Talks

Learnings from a Culture First Startup
CTO @Buffer
Sunil Sadasivan
A Microservice Based Architecture for Stream Processing using Spring Cloud Data Flow
Senior Staff Engineer @Pivotal
Mark Pollack
Becoming an Outlier
Software Architect @VinSolutions, Author @pluralsight
Cory House
ESPN Next Generation APIs Powering Web, Mobile, TV
Senior Director of Distribution Platforms @ESPN
Manny Pelarinos
WebSockets, Reactive APIs and Microservices
Ex-NASA researcher, Consultant
Todd Montgomery
Algorithms for Animation
Partner & Tech Lead @CarbonFive
Courtney Hemphill
Machine Learning Fast and Slow
Lead Data Scientist @betaworks
Suman Deb Roy
Vowpal Wabbit a Machine Learning System
Leading Machine Learning Researcher, Vowpal Wabbit Contributor
John Langford
Offense At Scale
Director - Penetration Testing / Red Team @Yahoo
Chris Rohlf

Tracks

Monday, 13 June

Tuesday, 14 June

Wednesday, 15 June

See the Full Schedule

Conference for Professional Software Developers

Follow QCon

Contact

Menu

QCons around the World

Qcon