ab-test-setup

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (usually 95%), and power (usually 80%). Use the quick reference tables in the sample size guide or online calculators like Evan Miller's to find the required visitors per variant.

Run the test until you reach the pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Avoid running longer than 4-8 weeks due to novelty effects and external factors, and never stop early based on peeking at results.

A/B tests compare two versions with a single change, A/B/n tests multiple variants requiring roughly 1.5-2x more traffic, and multivariate tests combine multiple element changes simultaneously requiring very high traffic. Choose based on traffic availability and how many changes you need to evaluate.

Early significance is often misleading due to the peeking problem, where checking results before reaching sample size inflates false positive rates. Day-of-week effects and audience mix shifts also distort early data, so commit to the full pre-calculated duration.

Define three tiers: a primary metric tied directly to your hypothesis, secondary metrics that explain why the change worked, and guardrail metrics that ensure nothing important gets worse. This structure prevents cherry-picking and protects against harmful side effects.

Skip testing when traffic cannot reach the required sample size within a reasonable duration, when the change is too small to detect, or when the risk of a bad variant is unacceptable. Alternatives include sequential testing methods, testing higher-traffic funnel stages, or using qualitative research instead.