helix

Start by sharpening the question into a testable claim, state the hypothesis and null, and define what result would falsify it. Then choose a design (RCT, A/B, factorial, observational, or ablation), control confounds, estimate sample size, and pre-specify the analysis before collecting data.

Sample size depends on the effect size you care about, desired power, and significance threshold. For a medium effect (d≈0.5) at 80% power you need roughly 64 per group; small samples like 15 per group only detect very large effects and risk false nulls.

A confound is a variable correlated with your treatment that could explain the result instead of your hypothesis. Control it through randomization, matching, adding control arms, or measuring it directly; anything you cannot control must be declared as a limitation.

Use ablation when you want to know which components of a system matter: remove one component at a time from the full system and measure the performance drop. Use A/B tests when comparing two complete variants against each other on a single outcome.

Deciding the test after seeing data enables p-hacking, even unintentionally, since enough post-hoc tests will always find something significant. Locking the primary outcome, test, threshold, and multiple-comparison corrections in advance keeps the result interpretable.

Use the best feasible alternative, such as a matched observational design with propensity matching or difference-in-differences to approximate a control group. The causal claim is weaker, so state exactly what the design can and cannot license.