reproduction-audit

Audits paper-reproduction pipeline reports and issues per-experiment reproduction verdicts.

Updated Aug 14, 2026
One-click install
npx skills add https://github.com/yyccbb/PaperReproduction-Skills --skill reproduction-audit-yyccbb
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: reproduction-audit
Source: https://github.com/yyccbb/PaperReproduction-Skills/tree/main/skills/reproduction-audit
Command: npx skills add https://github.com/yyccbb/PaperReproduction-Skills --skill reproduction-audit-yyccbb

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? After running a multi-stage pipeline to reproduce a machine learning paper, no single stage can judge whether the whole process held together or explain why reproduced numbers differ from the paper's. This Skill reads every stage's report, cross-checks claims against files on disk, and delivers a defensible verdict. ## Core Features & Use Cases - End-to-end evidence audit: Reads all stage reports under .paper-reproduction/, checks each against its own contract, and audits the seams between stages (verdict continuity, placeholder resolution, command fidelity, code state). - On-disk spot-checks: Verifies that cited run directories, run.json files, commit hashes, and analysis data actually exist and agree with what the reports claim. - Gap attribution and verdicts: Explains each paper-vs-reproduced delta using an eight-cause taxonomy, then issues per-experiment verdicts (REPRODUCED through NOT REPRODUCED) plus an overall evidence-chain rating. - Use Case: After stages 1-6 of a paper reproduction finish, run this audit to learn that Experiment 2's gap traces to a learning-rate drift between validation and the full run, and that the evidence chain is SOUND WITH GAPS. ## Quick Start Audit the completed reproduction in this repository and write the verdict report to .paper-reproduction/reproduction-audit.md.

Frequently Asked Questions about reproduction-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit a machine learning paper reproduction end to end?▼

Run the audit from the cloned paper codebase after earlier pipeline stages have written reports under .paper-reproduction/. It reads every stage report, cross-checks claims against files on disk, and writes reproduction-audit.md with per-experiment verdicts.

How to explain why reproduced results differ from the paper?▼

The audit attributes each gap using an eight-cause taxonomy covering protocol shortfalls, hyperparameter divergence, code fixes, environment drift, data differences, metric aggregation, paper ambiguity, and extraction error. Each cause must be tied to evidence in an upstream report.

Can the audit run if some pipeline stages never completed?▼

Yes. Missing upstream reports are recorded as findings rather than blockers. The audit reports which stage the pipeline reached and marks experiments without reproduced numbers as NOT REPRODUCED with the upstream verdict quoted verbatim.

Does the reproduction audit re-run experiments or analysis scripts?▼

No. It is strictly read-only: it never downloads, installs, edits code, or launches runs, and it never re-executes analysis scripts. A number that cannot be corroborated by reading is reported as a finding routed back to the owning stage.

What makes a reproduction verdict inconclusive?▼

A verdict is INCONCLUSIVE when the evidence chain is compromised, such as a command that drifted between validation and the full run, a missing run directory, or a run.json contradicting its report row. Matching numbers from unverifiable evidence never count as reproduced.