ab-test-designer

Structure the hypothesis as: if we make a change, then a metric will improve by an amount, because of a reason. The Skill formats your idea into this falsifiable statement and grounds it in context from your product and persona files.

Sample size depends on baseline conversion rate, minimum detectable effect, significance level (typically 95%), and power (typically 80%). The Skill uses these inputs with a standard calculator to produce required sample per variant and estimated test duration.

Use an A/B test when you can randomly split web or app traffic and need statistical significance. For pricing changes, feature betas, pilots, or fake doors where random assignment is impossible, use a general experiment design approach instead.

Define one primary metric that determines success, secondary metrics to watch, and guardrail metrics that must not degrade. Multiple primary metrics create p-hacking risk, so the plan enforces a single primary.

Underpowered tests lack enough samples to detect the target effect, so real improvements appear insignificant. Calculating sample size before launch ensures the test can actually detect the minimum effect worth acting on.