High Cardinality War Stories

High Cardinality – a term that can derail engineering leaders and software engineers alike. When seeking profound insights from their systems, they often find themselves limited by the cardinality constraints of their observability tools. But why should high cardinality matter, and how has it become an unavoidable challenge when it comes to monitoring software systems on a massive scale? Join me in this captivating talk as we delve deep into the concept of high cardinality, peeling back its layers to reveal its true essence. We'll start by dissecting metrics, understanding their vital role in the observability stack, and using their very structure to lay the groundwork for comprehending high cardinality. Prepare to witness how label sets exponentially multiply before our eyes, driven by our relentless pursuit of answers regarding system behavior and the dynamic nature of metrics. Can we simply turn a blind eye to cardinality and find happiness? I will show you why cardinality has become an unavoidable element in today's software engineering landscape, drawing from my own war stories in the field. From the battles against Noisy Neighbors to the epic Streaming Wars and the pulse of High Cardinality, we'll explore numerous problems that high cardinality can solve. But do our existing observability systems truly support handling high cardinality metrics at the scale we demand without exploding costs? Fear not, as we embark on an exploration of cutting-edge technologies that hold the answers to this burning question. Brace yourself for an enlightening journey that will transform the way you perceive high cardinality and your approach to observing software systems in the modern age.