What problem does it solve? Benchmarks and A/B evaluations often leak bias through leading prompts, non-equivalent treatment arms, shared-failure-mode oracles, or analysis rules chosen after seeing results. This Skill enforces a preregistered, falsifiable experiment design so benchmark outcomes can actually inform decisions. ## Core Features & Use Cases - Preregistration Manifest: Defines a JSON schema for decisions, claim boundaries, arms, hypotheses, metrics, controls, stopping rules, and oracle independence, validated by a bundled Python script. - Bias Stop Conditions: Blocks execution when arms differ beyond the declared treatment, prompts reveal the expected winner, or infrastructure failures would be scored as wrong answers. - Methodology Reference: Covers hypothesis registration, arm equivalence, neutral prompting, independent oracles, controls, failure handling, variance analysis, and reporting of nulls and misses. - Use Case: Before comparing two prompt formats across models, produce a validated manifest with opaque arm labels, a null control, an independent oracle with discrimination tests, and a frozen analysis plan timestamped before any results are collected. ## Quick Start Use the design-unbiased-benchmarks skill to design a preregistered A/B benchmark comparing two model configurations before running any evaluation.
