Workshop: Agile Streams With Apache Kafka

Apache Kafka is a de facto standard streaming data processing platform, being widely deployed as a messaging system, and having a robust data integration framework (Kafka Connect) and stream processing API (Kafka Streams) to meet the needs that commonly attend real-time message processing. In 2017, Confluent open sourced KSQL. A declarative, SQL-like stream processing language that lets you define stream processing applications easily. This allows rapid development of streaming applications and—more importantly—rapid iteration and improvement.

In this workshop we will explore best practices and architectural patterns of modern data integration with Apache Kafka and its ecosystem. This is one of the main use cases of Apache Kafka, and the use of Kafka allows integrating data systems and microservices in completely new ways. This leads to more performant, flexible, and robust data integrations. The workshop will combine theoretical discussion of best practices, lessons learned, and architecture patterns that we found useful, together with hands-on experimentation with a variety of projects from the Apache Kafka Ecosystem.

What you'll learn:

• Apache Kafka basics
• Data modeling for Apache Kafka
• Use of Schemas and Schema Registry
• Importance of Stream-Table duality for data integration
• Stream enrichment and Stream-Join patterns
• Rapid development of stream processing with KSQL
• Power and Flexibility of Kafka’s Streams APIs
• Hipster Stream Processing
• Best practices for taking data pipelines to production
• Common mistakes to avoid