Most applications continuously transform streams of inputs into streams of outputs. Yet the idea of directly modeling stream processing in applications is just coming into it's own after a few decades on the periphery.

This talk will cover the basic challenges of reliable, distributed, stateful stream processing. It will cover how Apache Kafka was designed to support capturing and processing distributed data streams by building up the basic primitives needed for a stream processing system. It will also introduce Kafka Streams, a lightweight library that applications can embed for expressing stateful stream processing operations.

Finally it will explore how Kafka and Kafka Streams solves practical problems in building scalable and stateful microservices, based on our experience building and scaling Kafka to handle streams that captured hundreds of billions of records per day.