measure-experiment-design

Start with a falsifiable hypothesis in the format "We believe [change] for [users] will [outcome] as measured by [metric]." Then define control and treatment variants, select one primary metric, calculate sample size, estimate duration, and set success criteria before launching.

Determine your minimum detectable effect, then compute the users needed per variant using a significance level of 0.05 and statistical power of 0.80. Document all assumptions so the calculation can be reviewed and reproduced.

Use the structure: "We believe [change] for [users] will [outcome] as measured by [metric]." Keep one hypothesis per experiment; if testing multiple changes, run separate experiments to isolate effects.

Duration depends on required sample size divided by available traffic. Account for weekly behavior patterns and avoid ending mid-week if user behavior varies by day, ensuring the test captures full cycles.

Guardrail metrics are measurements monitored alongside the primary metric to catch unintended negative effects, such as performance degradation or churn. They protect the product while you optimize for the primary outcome.

Avoid A/B tests when traffic is too low to reach adequate sample size, when the change cannot be isolated to a single hypothesis, or when qualitative research would answer the question faster than quantitative validation.