Presentation: Getting More Sleep, One SQS Message At A Time
Abstract
SparkPost is the fastest-growing cloud email API service, providing developers with robust email delivery so their applications and websites can send and receive email. SparkPost’s technology delivers over 25% of the world’s non-spam email. As you might expect, that means we generate a lot of data. Running a service has dramatically different requirements for scalability, reliability, and real-time performance. The tools and infrastructure that work well in the on-premises aren't always the ones best suited to the cloud. We knew we had to approach this from a different mindset than we might have in the on-prem past. When SparkPost shifted focus to cloud first my team started re-architecting problem areas to ensure better growth.
This talk will describe how my team transitioned from using single-node RabbitMQ message queues with clustered node.js ETL processes to embracing cloud technologies: Amazon Simple Queue Service(SQS), Elasticache(Redis backed), and reducing their node.js processes by 75% and transitioning to the “Omni ETL”, one ETL to rule them all. Moving to SQS gave us more breathing room when downstream services were slow or even down. The math is pretty simple: these changes mean fewer pages to our on-call engineers. Fewer pages = more sleep. More sleep = happier engineers. Happier engineers = a better product.
It was extremely cost effective because removing RabbitMQ allowed us to eliminate costly disk volumes. It also allowed us to be more aligned for auto-scaled, independent service tiers. The time to market was also fast because we took the best of what we already had and just solved the major pain points. We will also look at how we changed the engine of a 747 while in flight, and tips for you to do this too. Lastly we will share the outcome of this project and some lesson learned.
Similar Talks
Tracks
Monday, 26 June
-
Microservices: Patterns & Practices
Practical experiences and lessons with Microservices.
-
Java - Propelling the Ecosystem Forward
Lessons from Java 8, prepping for Java 9, and looking ahead at Java 10. Innovators in Java.
-
High Velocity Dev Teams
Working Smarter as a team. Improving value delivery of engineers. Lean and Agile principles.
-
Modern Browser-Based Apps
Reactive, cross platform, progressive - webapp tech today.
-
Innovations in Fintech
Technology, tools and techniques supporting modern financial services.
Tuesday, 27 June
-
Architectures You've Always Wondered About
Case studies from the most relevant names in software.
-
Developer Experience: Level up Your Engineering Effectiveness
Trends, tools and projects that we're using to maximally empower your developers.
-
Chaos & Resilience
Failures, edge cases and how we're embracing them.
-
Stream Processing at Large
Rapidly moving data at scale.
-
Building Security Infrastructure
How our industry is being attacked and what you can do about it.
Wednesday, 28 June
-
Next Gen APIs: Designs, Protocols, and Evolution
Practical deep-dives into public and internal API design, tooling and techniques for evolving them, and binary and graph-based protocols.
-
Immutable Infrastructures: Orchestration, Serverless, and More
What's next in infrastructure. How cloud function like lambda are making their way into production.
-
Machine Learning 2.0
Machine Learning 2.0, Deep Learning & Deep Learning Datasets.
-
Modern CS in the Real World
Applied, practical, & real-world dive into industry adoption of modern CS.
-
Optimizing Yourself
Maximizing your impact as an engineer, as a leader, and as a person.
-
Ask Me Anything (AMA)