product-data-metric-investigation-triage

Start with a 3-factor variance decomposition that splits the delta into volume, rate, and mix shift effects. Then slice the metric by platform, app version, geography, channel, and user cohort, and cross-reference the timeline with recent releases and feature flag changes.

Simpson's Paradox occurs when every individual segment's conversion rate improves but the blended top-line metric falls because traffic shifted toward lower-converting segments. It signals a mix shift problem in acquisition, not a product regression.

Use the formula delta R equals the sum of segment weight changes times average rates (mix effect) plus the sum of average weights times rate changes (rate effect). This quantifies exactly how many basis points came from traffic composition versus true conversion degradation.

Classify as P0 when the drop exceeds 20% in core conversion or revenue, is concentrated in a recent build or platform, and actively blocks users. Drops of 5-20% in a key funnel step are P1, while small mix-driven movements are P2.

Do not use this approach for infrastructure stack traces or crash logs, which belong to observability and debugging workflows. It also does not cover designing new North Star metric trees or running A/B test statistical analysis.