production-investigation

Start with a HEATMAP(duration_ms) and P99 query grouped by name, then run BubbleUp on the slow region to find differentiating fields like deployment version or endpoint. Drill into a slow trace with get_trace and verify the hypothesis with filtered queries.

BubbleUp compares a selected subset of events against the baseline to show which fields differentiate outliers. Run it via run_bubbleup with a query_pk and a 2D heatmap or group selection, then read dimension and measure charts ranked by differentiation.

Group COUNT by exception.message to categorize errors, then by service.name and http.route to find scope. Run BubbleUp comparing errored versus successful traffic, and inspect an errored trace with show_events enabled to see stack traces.

No signal usually means the selection is too broad or the anomaly is poorly isolated. Try a smaller time range, add filters to sharpen the outlier region, or select a different calculation before rerunning BubbleUp.

Look for spans with disproportionate duration versus their parent, sequential spans indicating N+1 patterns, error spans with span events, and gaps between child spans suggesting missing instrumentation or idle time.

You should not skip it, even when the cause seems obvious. BubbleUp routinely surfaces secondary causes, and verification queries protect against confirmation bias, which is strongest during incidents.