Presentation: What They Don’t Tell You About Microservices…

Location:

Duration

Duration: 
11:50am - 12:40pm

Day of week:

Level:

Persona:

Key Takeaways

  • Move past the hype and deep into the considerations needed to adopt Microservices.
  • Hear insights from a CTO building and deploying Microservices.
  • Learn about the process Yodle went through adopting and deploying microservices, including database architectures and architectural patterns that emerged.

Abstract

Who could say no to services that were fully testable, easy to scale, easy to re-write, composable, etc. You read a few books about micro-services and now you're hooked. Those blog posts by Martin Fowler made it sound so easy. Adrian Cockcroft’s videos were so convincing. All of the open source tools out there were going to make this a snap. But now you’re six months in and realize that it's not all so easy. We’ll discuss 6 things we wish we had known when we started our journey.

SPOILER ALERT: It is still worth it.

Topics we’ll cover:

  • Service Discovery and You
  • Databases
  • Testing; how hard can it be?
  • Dependencies shmependencies
  • Monitoring? Why would my monitoring be different?
  • Tools: your current ones have to evolve

Interview

Question: 
What is your role today?
Answer: 
I am the Chief Technology Officer of Yodle. That means I am responsible for every aspect of the technology that Yodle produces and that is both the software that we develop, the processes in which we deploy that to production, the processes through which we test it and uptime and maintenance of all the systems that it runs upon.
Question: 
What is the core work that Yodle does?
Answer: 
Yodle helps small businesses succeed in online marketing. We make it easy to grow and manage profitable customer relationships; it is success simplified.
We have a vertically-focused, operationally-embedded approach to digital marketing. We provide an online marketing solution including a CRM system so that small business owners can focus on their day-to-day craft while we automate and simplify their customer marketing. We have a portfolio approach aimed at the size of the business from the very small up to large national franchise businesses.
Question: 
What’s the motivation for your talk?
Answer: 
Microservices is a buzz word, and everybody wants to get on the microservices band wagon. It’s just that when you start going the microservices route, people don’t always realize that there are other things that have to happen, necessarily will happen, and it can quickly spiral out of control if you don’t stay on top of it. Yodle has a very smart group of people who are staying on top of the problem and helping evolve our infrastructure and our plans as we grow bigger in the microservices world.
Question: 
There are lots of definitions of microservices out there. What does a microservice mean to you?
Answer: 
I would like to say it’s something that has a very clear description. If you can say "it does x, y and z" in a simple sentence then perfect. If you need a full paragraph describing countless features then it’s probably not a microservice. It should be small. It should be re-buildable in a handful of days.
We have had examples of microservices handed off between one team and another, and for whatever reason, the new team wants to go and rewrite it. They should be able to just to do that and it shouldn’t be a big problem if they have chosen to do it, because these things are small enough.
That being said, we are not so picky because the way our PaaS is set up. We are decomposing a monolith into the same estate; we have some what I call macro-services: traditional SOA services that do 30, 40, 50 different things. I don’t think that is where we want to be in the long term, but it’s okay now, as long as you have sufficient testing and monitoring around those services so you can release them in a decoupled manner.
Question: 
Where do you consider the talk to be on the life cycle of adoption for micro services. At the beginning, middle, towards the end?
Answer: 
My goal is to take you on an actual journey: from October 2014 to today.
I am going to walk you through how the company had to evolve our technology as we went from having zero microservices to about 200 microservices today. Going from having three monolithically manually deployed service oriented architectures to where those are being decomposed and independently, automatically being deployed. We have about 60 more macro-services to migrate.
We quickly evolved into multiple codebases where there is a whole lot of tooling infrastructure needed around that. When you talk about how you are going to do continuous integration and continuous delivery, there is the testing infrastructure to think of. What has to happen to ensure consistent quality as you start breaking applications apart into smaller pieces? How can you the same quality as you push the product out in small batches?
Question: 
You mentioned databases earlier. Can you give me a taste of some of the things that you are going to talk about in the context of the database?
Answer: 
There are three things that we are primarily talking about. One is about shared data resources. We had a large PostgreSQL database. Prior to being dockerized microservices, everything was in Tomcat containers where we may have had perhaps 40 or 50 JVM’s hitting this database using connection pools. There might have been tons of services in one container there were sharing one connection pool.
As you start pulling services out, each new container has its own connection pool. At first, you are playing with connection pool management. You can adjust minimum idle and connection timeouts but ultimately, even the base load is too much for your database. If everybody is just sitting idle with 1 connection. With 200 micro-services running an average of 4 copies a piece, that is 800 connections to your database.
You could go to a minimum idle of zero but that is a questionable place to be for performance considerations. We switched to something called pgbouncer, which is a PostgreSQL library that acts as an external connection pool. We ran one of per slave host and all of the microservices are making connections to that, so essentially taking the connection pool and taking it out of the JVM.
The second one is about separation of data concerns: that is where you should have a dedicated database for every micro service. If you are going to start sharing data, you are going to have leaky abstractions. Other microservices are going to start picking things up out of your database and now services are starting to become tightly coupled and you don’t want to do that.
We had to build tooling to make it easier for the systems engineers to produce databases as quickly as they could to satisfy the needs of the developers. MongoDB databases with dedicated per service user accounts. PostgreSQL databases with per service user accounts tied to single schemas.
The third point on data services is identifying who is the troublemaker when looking at performance. If you are using a shared database, before we could track it back to a small set of hosts and have a pretty good idea of where usage was coming from. With containerized services that might be running 60 or 70 services per hosts. With appropriate monitoring you can see who is hogging the database resources with just at a glance.
Question: 
What are your key takeaways for this talk?
Answer: 
It’s not insolvable. Everything is built on trade-offs and you have to be willing to evolve as your systems evolve. There are a few things that you really need to know which is: "how am I going to keep the lights on?" and "how am I going to know when I have a problem?". Then finally, "how am I going to ensure quality?" Everything else is noise.
Question: 
QCon targets advanced architects and sr development leads, what do you feel will be the actionable things someone coming to your talk will leave with.
Answer: 
There will be lots of bumps along the way on your path to micro-services and that is OK and to be expected. You will need to iterate multiple times in expected and unexpected ways. Testing is one example. When you bring in micro services, especially if you are decomposing a monolith, you start getting into this situation where you are going to be changing your testing methodologies.
Initially you are working at the very fringes. But slowly you start testing things out and you get to the meat of your application. At this point you have to start thinking about dependency graphs between these services and how the application behave as a whole.
You need to ensure that your testing infrastructure can evolve with you. This is where having a way to express your dependencies, whether that be just something that your test engineers know, whether that is integrating into your CD pipeline, becomes important to you. So what we did was, we built a canary testing process that would allow developers to express dependencies and smoke tests in the application in the container on an endpoint named "tests". When a service gets deployed, before the service is available to route general traffic, the framework will ping the service to determine the tests, run them, validate them and finally decide: "okay, you are in place, you are good". Now the rest of the ecosystem can work. This could have been done just by expressing it in a file. Whatever you wanted to do and your test infrastructure has to work with that. This gives a way for developers to express dependencies and we can ensure that we are not breaking the ecosystem as we evolve.
We’ve done that and are now iterating to make that better using more advanced testing libraries like pact-jvm which can be more robust.
Question: 
What do you feel is the most disruptive tech in IT right now?
Answer: 
I think the most disruptive thing is actually not tech, the most disruptive thing is conferences and meetups. There has been an explosion in the amount of meetups, conferences and blog posts, way more than we have had in the past. It used to be that people would write books and that was the way you got the information out. Then it was journal articles.
Now everybody can publish everything, and there is an overwhelming deluge of information.
The good thing is, everybody is sharing everything, and that is allowing you to go so much faster from an innovation standpoint, because if I tell you about my infrastructure, and you throw a few rocks at me and tell me where you think it’s falling down, I am going to bring that back and make it better.
You are going to take those ideas and bring it into your company, and use those to make yourself better and then you are going to share that back out and that is a good feedback loop.
This way I don’t have just ten engineers on my payroll workin on a problem. I have 1000 engineers that are working on the exact same problem. I think it’s not tech, it’s the empowered communication that is disruptive but in a good way.

Speaker: Daniel Rolnick

CTO @Yodle

As Chief Technology Officer at Yodle, Daniel is responsible for all of the company's key technology initiatives. Daniel has more than 15 years of experience in engineering and architecture, with a strong background in leading impactful technology initiatives through their entire development life cycle. Daniel initially joined Yodle as Chief Architect before being promoted to Chief Technology Officer and becoming a member of the leadership team. Prior to working at Yodle, Daniel was Vice President of Research IT, POINT, for Barclays Capital. He was responsible for directing and leading the architecture for part of the company’s application infrastructure, among other things. Daniel has also held senior roles at Lehman Brothers – prior to being acquired by Barclays Capital – and UPOC Networks, an SMS text messaging network. Daniel received a B.S. in Computer Science from the Georgia Institute of Technology.

Find Daniel Rolnick at

Tracks

Monday, 13 June

Tuesday, 14 June

Wednesday, 15 June