experimental-design

Run a power analysis with your baseline rate and minimum detectable effect at alpha .05 two-sided and 80% power. The companion script computes it directly: python3 scripts/power.py n-props --p1 0.10 --p2 0.12 returns 3,841 users per arm for a 10% to 12% lift.

Preregistration means timestamping the hypothesis, primary metric, MDE, sample size, alpha and power, randomization mechanism, exclusions, analysis plan, and stop rule before any data arrive. A dated file or commit hash suffices, turning later changes into a visible diff.

The minimum detectable effect (MDE) is the smallest effect that would change a decision, chosen from decision stakes and prior evidence rather than from the available sample size. If the required n for that MDE is infeasible, accept a larger MDE, get more units, or drop the causal claim.

Peeking with an ad-hoc stop inflates false positives severalfold because each look is another chance to cross the threshold. The design must fix the horizon in advance or use a named sequential method, decided before launch.

Do not use it when data already exist (use significance testing instead), when reviewing someone else's finished trial (use risk-of-bias assessment), or when manipulation is impossible or unethical, where quasi-experimental designs like difference-in-differences apply.

No, scripts/power.py uses only the Python standard library and runs on Python 3.9+. It implements Cohen (1988) normal approximations and the Fleiss, Levin and Paik (2003) two-proportion formula, with a --selftest flag checking 56 values against published tables.