research-run

Pre-registration records five commitments before any numbers exist: the prediction with direction and size, the decision rule linking results to actions, the falsifier, the analysis path, and the search bounds. Split it into design quantities fixed before training and method hyperparameters fixed before touching test data.

For a two-arm comparison at conventional power, use roughly n ≈ 16σ²/Δ² per arm, where σ is pilot per-seed variance and Δ is the minimum difference worth acting on. Pairing arms on the same seeds usually requires far fewer runs than unpaired arms.

A non-significant result from an underpowered design is inconclusive, not evidence of absence. Check whether the design was ever powered for the minimum effect worth acting on; below that sample size the run cannot distinguish no effect from insufficient data.

Large per-seed differences despite shared seeds indicate the arms consume the RNG stream differently, so pairing only held nominally. Diagnose this by reading the arms' RNG consumption in code, not the numbers, since it looks identical to a heavy-tailed effect.

Distrust results when the best cell was selected from many unnamed comparisons, when arms are saturated at a floor or ceiling, when a searched hyperparameter landed at the grid edge, or when the baseline was reimplemented rather than the authors' reference code.