What problem does it solve? Vendor posts, papers, and release notes routinely assert benchmark numbers ("91% on GSM8K", "outperforms GPT-4o", "SOTA on MMLU") that collapse under scrutiny because of test-set contamination, unpinned baselines, single runs, self-evaluation, cherry-picked metrics, or underpowered test sets. This Skill turns such a claim into a structured audit so you know how much evidential weight the number actually carries before quoting it in a decision. ## Core Features & Use Cases - Six-domain integrity audit: Scores test-set contamination, baseline pinning, seed/run variance, evaluator independence, metric selection, and test-set size as Low / Some concerns / High risk, each with a one-sentence rationale. - Deterministic reliability score and risk tags: Computes reliability = max(0, 5 - 1xHigh - 0.5xSome concerns) and attaches machine-readable tags from a fixed vocabulary such as contamination-unverified, baseline-version-drift, and single-seed. - Supports / does-not-support split with replication plan: States the weaker claim the evidence sustains, the claim it invalidates (and the domain that kills it), and the specific experiment that would raise the score. - Use Case: A vendor blog claims "AcmeLM-2 scores 91.2% on GSM8K, outperforming GPT-4o" from one run against a year-old GPT-4o figure. The audit flags High risk on baseline pinning and run variance, scores the claim 2/5, and concludes it supports only "competitive with 2024-era frontier models" — not any state-of-the-art framing. ## Quick Start Use the benchmark-model-claims skill to audit this claim before we quote it: a vendor blog says their model scores 91.2% on GSM8K and outperforms GPT-4o based on a single run.
