What problem does it solve? Claims like "Model A scored 87% vs Model B's 85%" or "the new variant lifted conversion 12%" are routinely treated as real differences when they may be sampling noise — or dismissed when they are real. This Skill decides whether a reported gap between two groups is statistically distinguishable from noise, and forces the full picture: p-value, 95% confidence interval on the difference, and an effect size, never a bare verdict. ## Core Features & Use Cases - Test selection by data type: pooled two-proportion z-test, Fisher's exact for small counts, chi-square, Welch's t-test for unequal variances, and paired t-test for repeated measures, with a selection table in references/tests-and-effect-sizes.md. - Deterministic companion tool: scripts/significance.py (standard library only, Python 3.9+) computes exact p-values, confidence intervals, Cohen's h/d, and Hedges' g, with a 12-check self-test verified against published examples. - Guardrails built in: flags underpowered tests, multiplicity across many comparisons, unstated sample sizes (reported as unverifiable), and the anti-pattern of calling a non-significant result "equivalent". - Use Case: A vendor claims their model beat a competitor 87% to 85% on a 500-item benchmark. The Skill converts the rates back to counts, runs a two-proportion test, and shows the gap is within sampling noise — without declaring the models equivalent. ## Quick Start Ask the AI to run a significance test on your two-group comparison, for example: "Control converted 96 out of 800 and treatment 128 out of 800 — is this A/B result significant, and what is the confidence interval on the difference?"
