Spotify’s data is increasing at a rate of 60 billion events per day. The current event delivery system, which is based on Kafka 0.7, is slowly but certainly reaching its limitations. To be able to seamlessly scale the event delivery system with Spotify’s growth, we decided to base the new event delivery system on Google Cloud Pubsub and Google Cloud Dataflow.

Spotify’s event delivery system is one of the foundational pieces of Spotify’s data infrastructure. It has a key requirement to deliver complete data with a predictable latency and make it available to Spotify developers via well-defined interface. Delivered data is than used to produce Discover Weekly, Spotify Party, Year in music and many other Spotify features.

This talk is going to cover the evolution of Spotify’s event delivery system focusing on the lessons learned from operating it and reasons behind the decision of moving it into the cloud.

This talk will also going to cover Scio, which we developed to make the Dataflow SDK easier to use. Scio is a high level Scala API for the Dataflow SDK which brings Dataflow closer to popular data processing frameworks.