You are viewing content from a past/completed QCon

Presentation: Using AI to Optimize SQL Query Plans and Performance

Track: Sponsored Solutions Track III

Location: Liberty, 8fl.

Duration: 2:55pm - 3:45pm

Day of week: Wednesday

Share this on:


Digital transformation is enabling organizations to discover valuable new ways to leverage structured and unstructured data. Queries are very powerful since they specify what information is desired from a data source. Applications issue queries to read and write tables that transform and summarize Big Data into actionable information and insights. Queries using the high level SQL language and Hive let the query engine build an execution plan, avoiding the complexity of MapReduce and promising greater efficiency and faster results with less effort.

However, cluster resources are not unlimited, and at some point the operator (and developer) will observe a query that is running more slowly than usual, or has stopped altogether. An inefficient query may pose a burden on the database’s resources, cause slow query performance, or even impact other cluster users if the query contains errors. Poor performance can result from many sources, including CPU and memory bottlenecks, an excessive number of rows, and failed stages, forcing the query operator to intervene. But without clear visibility into the query plan, resolving these issues can prove to be difficult and time-consuming.

In this informative session, Pepperdata field engineer Kirk Lewis will discuss how Pepperdata’s query performance management solution, Query Spotlight, uses AI to continuously monitor and analyze queries, providing operators with a top-down view of cluster resources combined with detailed insight into query plans, including duration time, memory/CPU usage, stages, and critical path for every query. Learn why the real-time correlation of application and infrastructure performance is essential to optimizing query plans and ensuring efficient cluster operations.

Speaker: Kirk Lewis

Sr. Field Engineer @pepperdata

Kirk joined Pepperdata in 2015. Previously, he was a Solutions Engineer at StackVelocity. Before that he was the lead technical architect for big data production platforms at American Express. Kirk has a strong background in big data.

Find Kirk Lewis at


  • Blameless Culture

    Absorb the lessons learned from failures and outages in a human-centric process.

  • Modern CS in the Real World

    Learn how companies are applying recent CS research to tackle concurrency, distributed data, and coordination.

  • Architectures You’ve Always Wondered About

    Next-gen architectures from the most admired companies in software.

  • Bare Knuckle Performance

    Learn from practitioners on the challenges and benefits of architecting for performance and much more.

  • Java - The Interesting Bits

    Learn the new features in the recent and near-future releases of Java and the JVM and what they offer.

  • Ethical Considerations in Consciously Designed Software

    Design considerations for various contexts, locations, security and privacy requirements.

  • Operating Microservices

    Learn from practitioners operating and evolving systems in performance demanding environments.

  • Security

    Learn how to make security an inherent part of the software development process.

  • Native Compilation Is Back (A Look at Non-Vm Compilation Targets)

    Issues with native compilation for in browser-based and server-side environments

  • Trouble-Shooting in Production

    Learn debugging strategies for complex and high stakes environments where standard debuggers and profilers fail.

  • Predictive Architectures and ML

    Explore the systems and designs covering the full loop from machine learning to inferencing.

  • Data Engineering on the Bleeding Edge

    Explore the latest trends in data engineering that help improve the life of the developer

  • Production Readiness

    Observability, emergency response, capacity planning, release processes, and SLOs for availability and latency.

  • Humane Leadership

    A look at leadership with an emphasis on empathy, taking chances and building other leaders within organizations and teams

  • Devex & Teams

    Explore how to reduce developer friction between teams and stakeholders.