Since it was open sourced, Apache Kafka has been adopted very widely from web companies like Uber, Netflix, LinkedIn to more traditional enterprises like Cerner, Goldman Sachs and Cisco. At these companies, Kafka is used in a variety of ways - as a pipeline for collecting high-volume log data for load into Hadoop, a means for collecting operational metrics to feed monitoring and alerting applications, for low latency messaging use cases and to power near realtime stream processing.

Kafka’s unique architecture allows it to be used for real time processing as well as a bus for feeding batch systems like Hadoop. Kafka is fundamentally changing the way data flows through an organization and presents new opportunities for processing data in real time that were not possible before. The biggest change this had led to is a shift in the way data is integrated across a variety of data sources and systems.

In this talk, I will discuss how companies are using Apache Kafka and where it fits in the Big Data ecosystem.