statistical-analyst

Run a two-proportion Z-test for conversion rates or Welch's t-test for continuous metrics using the hypothesis tester script with your control and treatment sample sizes and observed values. It returns the p-value, confidence interval, effect size, and a plain-English verdict on whether to ship.

Use the sample size calculator with your baseline conversion rate, minimum detectable effect as a relative change, significance level, and desired power. It returns the required n per variant, total sample, and a duration estimate if you provide daily traffic.

Statistical significance (p < alpha) means the difference is unlikely due to chance, while practical significance depends on effect size. A result can be significant with a negligible Cohen's h or d, meaning the business impact is too small to justify shipping.

No, all scripts use only the Python standard library math module. Normal, t, and chi-square distributions are computed via erfc, continued fractions, and series expansions, so no external packages need installation.

Avoid them with fewer than 30 samples per group without normality checks, heavy-tailed metrics like revenue with outliers, sequential peeking scenarios, or clustered data where observations are not independent. Use sequential testing, non-parametric methods, or cluster randomization instead.

Peeking before the planned end date inflates the effective alpha — checking at 50%, 75%, and 100% of planned n raises the true false positive rate from 5% to roughly 13%. Pre-commit to a stopping rule or use sequential testing methods like SPRT.