experiment-design-and-prioritization

Use ICE scoring: rate each hypothesis on Impact, Confidence, and Effort from 1 to 5, then sum them. Tests scoring 12 or higher are Priority 1. Ground Impact and Confidence scores in analytics data like page sessions, RPS, and funnel abandonment rather than intuition.

At 95% significance and 80% power with a 50/50 split, detecting a 5% lift needs about 10,000 sessions per variant, 10% needs about 2,500, and 20% needs about 650. Low-traffic sites should test larger changes with a higher minimum detectable effect.

A good hypothesis names the specific element changed, the target page, the primary metric (RPS), the audience, the expected direction and magnitude of change, and cites at least one research data source. "Try a better hero image" is a direction, not a testable prediction.

Stop early only when a predefined Do No Harm guardrail triggers, such as RPS dropping more than 15% below baseline with over 80% confidence the result is negative. Normal variance in the first week often shows negative swings that correct by day 14.

Start with a blanket test for new hypotheses when traffic supports multi-segment analysis within two weeks. Use a targeted test when one segment holds over 50% of sessions with significantly lower RPS, or when a prior blanket test showed a clear segment-specific effect.