split-test-interpretation

A test needs at least 95% statistical significance and 80% statistical power, determined by session volume and effect size. It must also run a minimum of two weeks regardless of apparent early results, since day-of-week and seasonal patterns can create false lifts.

Use the ICE method: score each hypothesis 1–5 on Impact, Confidence, and Effort, then sum the scores. Run the highest-scoring tests first and update scores after each concluded test, since every result changes the confidence of related hypotheses.

Segment the result by the four traffic Quadrants using new_user and device filters before calling it inconclusive. A flat aggregate often hides a strong lift in one segment, which justifies a Targeted rollout to that segment only.

Sites under 10,000 sessions per month often cannot reach statistical validity in a reasonable window. Use live testing instead: ship the change, run 30–60 days, and compare against the pre-change baseline, treating results as hypothesis generation rather than validation.

Run a maximum of two simultaneous tests per user journey, where a journey spans from ad click to checkout. More concurrent tests create attribution distortion that makes every individual result unreliable.

The lift may have been a time artifact from seasonality or ad spend changes. Re-measure the winning segment 30 days post-implementation; if the lift evaporated, consider a reverse A/B test reintroducing the previous version.