What problem does it solve? AI assistants routinely set up A/B tests that declare winners on tiny samples, stop at the first p < 0.05, optimize one metric with no guardrails, and end at "B won" with no learning loop. This Skill enforces statistical discipline so experiments produce trustworthy decisions instead of confident false conclusions. ## Core Features & Use Cases - Pre-registration discipline: Writes falsifiable if/then/because hypotheses with success and failure criteria plus pre-set iterate/pivot/persevere thresholds before the test runs. - Correct sizing and execution: Calculates sample size from baseline, MDE, and confidence level, derives duration, enforces a full business cycle, and prevents peeking and mid-test split changes. - Honest measurement and hand-off: Uses one primary metric plus guardrails, reads segments and long-term holdouts, documents results, and routes to metric-validity and assumption-rigor audits. - Use Case: Ask it to design an A/B test for a new checkout flow, and it will first check traffic sufficiency, compute the required sample size, set a 99% confidence threshold given the risk, define guardrail metrics like revenue and support volume, and pre-commit decision thresholds. ## Quick Start Ask the assistant to design an A/B test for your proposed change, including hypothesis, sample size, duration, primary and guardrail metrics, and decision thresholds.
