What problem does it solve?
This Skill helps you avoid false conclusions when debugging flaky tests, noisy metrics, intermittent failures, and subtle regressions. It turns one-off observations into structured probabilistic analysis so you can decide whether a change is real, random, or still unproven.
Core Features & Use Cases
- Flake and noise analysis: Quantifies failure rates, sample sizes, confidence, and base rates instead of treating single runs as proof.
- Experiment interpretation: Evaluates A/B-style outcomes, deploy effects, latency shifts, and conflicting CI results while accounting for variance and confounders.
- Bias and confounder detection: Surfaces selection bias, survivorship bias, regression to the mean, non-stationarity, and observer effects before you attribute causality.
- Use case: When a test passes after a suspected fix but has failed intermittently in the past, this Skill guides you to measure the baseline, collect enough runs, and report whether the observed improvement exceeds the expected noise floor.
Quick Start
Use the statistical-debugging skill to assess whether a flaky CI test is actually fixed or whether the latest green runs are still consistent with its historical failure rate.
