ab-testing

Sample size depends on your baseline conversion rate, minimum detectable effect, 95% significance level, and 80% statistical power. Use the quick-reference tables in the sample size guide or calculators like Evan Miller's to find the required visitors per variant.

Run until you reach the pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Avoid running longer than 4-8 weeks due to novelty effects and external factors, and never stop early based on peeking at results.

Early significance is often misleading due to the peeking problem, regression to the mean, and audience mix shifts. Checking results before reaching the planned sample size inflates false positive rates, so commit to the full duration or use sequential testing methods.

A/B tests compare two versions with a single change, while multivariate tests evaluate combinations of multiple elements simultaneously. MVT requires dramatically more traffic since combinations multiply, so sequential A/B tests are recommended for lower-traffic pages.

Define three tiers: a primary metric tied directly to your hypothesis, secondary metrics that explain why the change worked, and guardrail metrics that must not get worse. This prevents shipping wins that quietly harm other parts of the business.

Skip testing when traffic cannot reach the required sample size within a reasonable duration, when the change is too small to detect, or when the risk of a bad variant is unacceptable. In those cases, rely on qualitative research or bolder single changes.