Lessons learned building Storm
Location:
Grand Ballroom - Salon A/B
Track:
Abstract:
Storm is an open-source, distributed, realtime computation system. Storm is a multi-tenant system in which diverse and independent applications share the same resources on the cluster. Building Storm involved numerous challenges related to distributed systems, resource scheduling, resource contention, and capacity planning. We solved these problems by relying on a simple motto: the more complex the problem, the simpler the solution must be. In this talk you'll see what we learned building out Storm, both from a technical and process standpoint.
These lessons are hugely relevant to anyone working on complex systems or building out shared infrastructure.
Nathan Marz is currently working on a new startup. Previously, he was the lead engineer at BackType before being acquired by Twitter in 2011. At Twitter, he started the streaming compute team which provides and develops shared infrastructure to support many critical realtime applications throughout the company. Nathan is the creator of many open source projects which are relied upon by over 50 companies around the world, including projects such as Cascalog and Storm. Nathan is also working on a book for Manning publications entitled "Big Data: principles and best practices of scalable realtime data systems" due out for publication later this year.