ab-test-setup

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (usually 95%), and power (usually 80%). Use the quick reference tables in the sample size guide or online calculators like Evan Miller's to find the required visitors per variant.

A/B tests compare two versions with a single change, A/B/n tests multiple variants requiring roughly 1.5-2x more traffic, and multivariate tests combine multiple element changes simultaneously requiring very high traffic. Choose based on your traffic volume and how many changes you need to evaluate.

Stopping early after peeking at results inflates false positive rates and leads to wrong decisions. Pre-commit to your calculated sample size and duration, or use sequential testing methods offered by tools like Optimizely or VWO if early stopping is necessary.

Use a three-tier framework: one primary metric tied directly to your hypothesis, secondary metrics that explain why the change worked, and guardrail metrics that catch negative side effects. For example, a pricing test might track plan selection rate, time on page, and refund rate.

Increase your minimum detectable effect to 20% or more, reduce the number of variants, test higher-traffic pages earlier in the funnel, or combine traffic across similar pages. If none work, rely on qualitative research instead of running an underpowered test.