warning icon QCon New York 2020 has been canceled. See our current virtual and in-person events.
You are viewing content from a past/completed QCon -

Presentation: Index-Free Log Aggregation With Kafka

Track: Sponsored Solutions Track II

Location: Liberty, 8fl.

Duration: 1:40pm - 2:30pm

Day of week: Tuesday

Slides: Download Slides


We are seeing a rise in a new generation of logging solutions. Traditional approaches have been based on database indexes, but the next wave of log management platforms is a combination of techniques including streaming queries, in-memory processing, brute-force search, and filtering.  We call this *Index-Free Logging*.

In this talk, we will provide an overview of the advantages of Index-Free Logging, and how we’re implementing it in Humio, a new log analytics system with a novel storage and query engine for events logs and traces.  Humio was built from the ground up as a multi-tenant on-prem system so you can install it behind your own firewalls or in your own cloud infrastructure.

In a Humio setup, we then use Kafka for two things: buffering ingest and as a sequencer of events among the nodes of a Humio cluster. We love Kafka — it has made implementing a distributed system significantly easier than if we had to do the distributed queue from scratch. Using Kafka as a building block has allowed us to focus our attention on building a fast and easy search experience.

Speaker: Kresten Krab Thorup

CTO @MeetHumio (Humio)

Kresten provides technical leadership and vision at Humio. In his previous role as CTO of Trifork, Kresten was responsible for technical strategy and has provided consulting advice to teams on a variety of technologies to include distributed systems and databases, erlang, Java, and mobile application development. Kresten has been a contributor to several open source projects, including GCC, GNU Objective-C, GNU Compiled Java, Emacs, and Apache Geronimo/Yoko. Prior to Trifork, Kresten worked at NeXT Software (now acquired by Apple), where he was responsible for the development of the Objective-C tool chain, the debugger, and the runtime system. Kresten has a Ph.D. in computer science from University of Aarhus.

Find Kresten Krab Thorup at