Conference: Jun 26-28, 2017
Workshops: Jun 29-30, 2017
Presentation: Presidential Campaigns & Immutable Infrastructure
What You’ll Learn
- Understand how to build systems that are designed to fail in graceful ways.
- Hear stories of rapid growth in very short periods of time.
- Learn techniques and practices to improve System Reliability Engineering.
Abstract
Hillary for America was arguably one of 2016’s largest startups. It was in the news every day, raised billions of dollars, and grew at an incredibly fast rate. There was even a very splashy exit. But what isn’t often talked about is the technical infrastructure behind it. Over the course of 18 months, HFA tech’s SRE team built and ran an immutable infrastructure, supporting a tech org that started with one developer and grew to 80, letting people deploy hundreds of times a day, with little to no downtime. In this talk Michael will explore how the campaign systematically approached every design decision to stay true to immutable principles, leveraging AWS infrastructure along with open source technology like Packer, Ansible, Consul, and a healthy dose of Varnish.
Interview
Michael: I joined Hillary for America at the beginning of a campaign in June of 2015. At that point, we had just a few things that we were doing. These were things like collecting money online, trying to get people to sign up for emails, or keeping engagement with web site.
From there, we built out some of the initial products the campaign had before moving on to creating a site reliability engineering team which handled build and deploy tooling. But, most importantly, we architected the more than 70 microservices that ran throughout the campaign.
All 70 of these Microservices were built with immutable infrastructure that did lots of interesting stuff. These were things that took money, signed people up for events, call tools, sync tools, voter protection tools, the list goes on and on.
By the end of the campaign, we were an SRE team of four and a tech team of 80 (including more than 50 software engineers pushing code every single day).
Michael: Basically, my motivation is to answer the question with 50 engineers pushing code as fast as the can and an SRE team of four, how do you do that?
How do you balance the needs of your developers against reliability? That's where immutable infrastructure came in. So I want to talk about how that's the handshake agreement between developers that are moving insanely fast and reliability concerns.
I want to talk about what our stack looked like, and how we did it. I will also talk about some of the tools we used like Consul and Vanish.
Michael: I'm talking to is somebody who is an architect, reliability engineer, or a person who is in the position of making decisions right. So they're not just implementing. They are making calls on how to prioritize what they're working on and the tradeoffs.
What I want them to come away with is an understanding that even with the best immutable infrastructure plan, you will fail. So the question is really about how you build an immutable infrastructure system that will scale. But, more importantly, allows you to fail gracefully and in a way that your users don't don't notice.
Similar Talks
Tracks
Monday, 26 June
-
Microservices: Patterns & Practices
Practical experiences and lessons with Microservices.
-
Java - Propelling the Ecosystem Forward
Lessons from Java 8, prepping for Java 9, and looking ahead at Java 10. Innovators in Java.
-
High Velocity Dev Teams
Working Smarter as a team. Improving value delivery of engineers. Lean and Agile principles.
-
Modern Browser-Based Apps
Reactive, cross platform, progressive - webapp tech today.
-
Innovations in Fintech
Technology, tools and techniques supporting modern financial services.
Tuesday, 27 June
-
Architectures You've Always Wondered About
Case studies from the most relevant names in software.
-
Developer Experience: Level up Your Engineering Effectiveness
Trends, tools and projects that we're using to maximally empower your developers.
-
Chaos & Resilience
Failures, edge cases and how we're embracing them.
-
Stream Processing at Large
Rapidly moving data at scale.
-
Building Security Infrastructure
How our industry is being attacked and what you can do about it.
Wednesday, 28 June
-
Next Gen APIs: Designs, Protocols, and Evolution
Practical deep-dives into public and internal API design, tooling and techniques for evolving them, and binary and graph-based protocols.
-
Immutable Infrastructures: Orchestration, Serverless, and More
What's next in infrastructure. How cloud function like lambda are making their way into production.
-
Machine Learning 2.0
Machine Learning 2.0, Deep Learning & Deep Learning Datasets.
-
Modern CS in the Real World
Applied, practical, & real-world dive into industry adoption of modern CS.
-
Optimizing Yourself
Maximizing your impact as an engineer, as a leader, and as a person.
-
Ask Me Anything (AMA)
