Presentation: Survival of the Fittest - Streaming Architectures

Track: Stream Processing at Large

Location: Liberty, 8th fl.

Duration: 5:25pm - 6:15pm

Day of week: Tuesday

Level: Intermediate

Persona: Architect, Data Scientist


​“Perfect is the enemy of good” ​ ​ -​ ​Voltaire

On the journey through life, we learn and adapt via trial and error - software development is no different. We realize and accept that we won’t build the perfect solution the first time around, it takes many iterations. At, now part of HBC Digital, we started processing and streaming event data nearly 5 years ago. Our initial solution was dramatically different from our current solution - and will likely be different from our solution 5 years from now.

The banner, at HBC Digital, is in the business of flash sales, which makes for some interesting use cases in the world of streaming. We release new sales of top designer labels, at up to 70% off retail, on the web and our mobile app, every day at Noon and 9pm. Around the time of these releases, we experience volume spikes between 10X and 100X on our streams.

Numerous streaming frameworks, homemade, as well as, open source, did not pass the evolutionary tests. Frameworks come and go, ​so this talk is not about the “best” framework or platform to use, rather it’s about core principles that will stand the tests of streaming evolution. Also, this talk covers major potential pitfalls that you may stumble over on your path to streaming, as well as, how to avoid these. Finally, this talk will cover what the next evolutionary step in streaming at HBC Digital. ​

Speaker: Michael Hansen

Principal Data Engineer @hbcdigital

Michael has two decades of technical and leadership experience in engineering big data, data warehouse, and data streaming systems, as well as, the full-stack environments, automation, and tooling surrounding these. Currently he is working as Principal Data Engineer at HBC Digital, responsible for all things related to data plumbing. He holds a Bachelor of Science in operations research and industrial engineering from UC Berkeley.

Find Michael Hansen at

Similar Talks

Software Engineer @Agrilyst
Cofounder & CTO, previously Co-Founder & CTO @Gilt
Platform Director, "SeatGeek Open"​ @SeatGeek
Director of Engineering @ Squarespace
Software Engineer @Jet, previous CTO
SVP Engineering, HBC Digital / Gilt & Committer Apache Karaf
Leading Machine Learning Researcher, Vowpal Wabbit Contributor


Monday, 26 June

Tuesday, 27 June

Wednesday, 28 June