Presentation: Getting More Sleep, One SQS Message At A Time

Track: Sponsored Solutions Track III

Location: Shubert - Uris, 6th fl.

Duration: 2:55pm - 3:45pm

Day of week: Wednesday


SparkPost is the fastest-growing cloud email API service, providing developers with robust email delivery so their applications and websites can send and receive email. SparkPost’s technology delivers over 25% of the world’s non-spam email. As you might expect, that means we generate a lot of data. Running a service has dramatically different requirements for scalability, reliability, and real-time performance. The tools and infrastructure that work well in the on-premises aren't always the ones best suited to the cloud. We knew we had to approach this from a different mindset than we might have in the on-prem past. When SparkPost shifted focus to cloud first my team started re-architecting problem areas to ensure better growth.

This talk will describe how my team transitioned from using single-node RabbitMQ message queues with clustered node.js ETL processes to embracing cloud technologies: Amazon Simple Queue Service(SQS), Elasticache(Redis backed), and reducing their node.js processes by 75% and transitioning to the “Omni ETL”, one ETL to rule them all. Moving to SQS gave us more breathing room when downstream services were slow or even down. The math is pretty simple: these changes mean fewer pages to our on-call engineers. Fewer pages = more sleep. More sleep = happier engineers. Happier engineers = a better product.

It was extremely cost effective because removing RabbitMQ allowed us to eliminate costly disk volumes. It also allowed us to be more aligned for auto-scaled, independent service tiers. The time to market was also fast because we took the best of what we already had and just solved the major pain points. We will also look at how we changed the engine of a 747 while in flight, and tips for you to do this too. Lastly we will share the outcome of this project and some lesson learned.

Speaker: Bob Evans

Director of Engineering - Applications

JavaScript, JavaScript, JavaScript. I’ve been mostly writing node.js for 4 years while building out the capabilities of SparkPost. I manage a growing team of 11 full-stack javascript engineers and work hard allowing developers and companies like Pinterest, ZipRecruiter, Zillow and the Financial Times send billions of messages a month. I’m obsessed with my Daily Mix 1 on Spotify and listen to bluegrass/jam at 11. Attempting to figure out this whole parenting of 2 thing and watching baseball when I have time, Go O’s.

Find Bob Evans at

Speaker: Jason Sorensen

Lead Data Scientist @SparkPost

Jason Sorensen is the Lead Data Scientist at SparkPost. Jason played a major role in architecting solutions for SparkPost's data pipeline. Jason has a Master's in Mathematics from the University of Minnesota and a Master's in Computer Science from Johns Hopkins University. In his free time Jason enjoys board games and cycling.

Find Jason Sorensen at

Similar Talks

Lead Software Engineer @ Capital One
Leads Product & Application Security @Netflix


Monday, 26 June

Tuesday, 27 June

Wednesday, 28 June