What problem does it solve? Agent-system claims are easy to assert and hard to falsify: LLM judges are anti-correlated with truth on false-success detection, benchmark scores swing 10-20 points on harness alone, and multi-agent wins are often just bigger token budgets. This Skill designs experiments that can actually kill a wrong thesis cheaply, and rejects designs that cannot. ## Core Features & Use Cases - Falsification-first design: Orders experiments so the cheapest test that can kill the thesis runs first, with kill criteria pre-registered before any data exists and never moved afterward. - Deterministic oracles and ceiling tests: Replaces LLM judges in the primary success path with SQL assertions, recording proxies, and reference implementations, and uses a hand-written ideal-tool arm (A8) to separate the value of the idea from the quality of the synthesizer. - Contamination and denominator audits: Detects evaluation sets scored against post-fix states, denominators padded with records the instrument declined, and ablations whose positive result is indistinguishable from a broken instrument. - Use Case: Before building a tool-synthesis pipeline, run the ceiling test comparing hand-written ideal tools against a shell-only baseline; if they score the same, the thesis is dead and it cost a week instead of a quarter. ## Quick Start Use the experiment-design skill to review my planned benchmark comparing a single agent against a multi-agent setup and tell me which controls and kill criteria are missing.
