Cribbs: With my background working on distributed databases I realized writing to disk first and then trying to process it afterwards is really expensive. It's fine for generating business intelligence afterwards, or if you want to aggregate ahead of time before you write to disk it's fine. But if you're trying to look at an individual, like a single customer who has had a bad experience, aggregates don't help you. You just want to look at the very focused events that are related to that person's bad experience. I think that stream processing is a really good fit for that sort of use case, finding the really tiny needles in a haystack.

So you have all this data flowing in: what you do with it, do you even know what's there? In some cases you do because you wrote their instrumentation for the application and it's emitting data like a request span with the latency measurement, but I think there is still a barrier to entry. Instead of thinking of it procedurally, you have to be thinking of it as a stream, and do things with windows and across different dimensions. It's an unfamiliar problem to a lot of people.

Cribbs: We have built up a lot of infrastructure in Elixir - it's the stuff that we're using under the hood, but we've built a pretty interesting DSL on top because we know that we can focus that on operational instrumentation data. At every stage of the stream we have a schema, so we can optimize around that.