gauge

Run both systems across multiple seeds (typically 5 or more) and report mean plus variance for each. A score gap is only real if it exceeds the run-to-run spread; a 3-point gain inside a 5-point variance is noise, not progress.

Choose the metric from the decision and its error costs: use precision/recall or AUPRC for imbalanced classes, calibration when probabilities matter, and cost-weighted metrics when error types differ in impact. Always state what the chosen metric ignores.

Contamination occurs when test data appears in a model's training set, inflating scores. For public benchmarks and pretrained models, assume contamination until ruled out, and use fresh, held-out, or private eval sets for claims that matter.

Validate the judge against human labels on roughly 150 examples before trusting it, and report the agreement rate. Randomize response order to control position bias, and watch for verbosity bias and self-preference toward its own model family.

An overall gain can come from the easy majority while a key subgroup or hard slice collapses. Always break headline metrics out by user segments and difficulty bands before declaring a win.

Distrust any score from one run of a stochastic system, any public benchmark a pretrained model may have seen, and any result without trivial and strong baselines. A reproducible harness with pinned seeds, prompts, and data versions is required for a falsifiable number.