paper-claim-audit

Verifies every numeric claim in a research paper against raw result files using a zero-context cross-model reviewer.

Updated Mar 23, 2026
One-click install
npx skills add https://github.com/hanasho744/codex --skill paper-claim-audit-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: paper-claim-audit
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/paper-claim-audit
Command: npx skills add https://github.com/hanasho744/codex --skill paper-claim-audit-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? When the same agent runs experiments and writes the paper, confirmation bias creeps in: numbers get rounded up, best seeds get reported as averages, and deltas get miscalculated. This Skill catches those discrepancies by auditing every quantitative claim in the paper against raw evidence with a fresh reviewer that has zero prior context. ## Core Features & Use Cases - Zero-Context Cross-Model Audit: Sends only paper .tex files and raw result files (JSON/CSV/YAML) to a fresh GPT-6-Astra reviewer thread, excluding all executor summaries, logs, and prior audit results to prevent confirmation bias. - Seven Failure-Mode Checks: Detects number inflation, best-seed cherry-picking, config mismatches, aggregation mismatches, delta arithmetic errors, caption-table mismatches, and scope overclaims. - Structured Verdict Artifacts: Always emits PAPER_CLAIM_AUDIT.json (with verdict, reason_code, SHA256 input hashes, and trace path) plus a human-readable Markdown report, with verdicts PASS, WARN, FAIL, NOT_APPLICABLE, BLOCKED, or ERROR. - Use Case: Before submitting a paper, run the audit to confirm the abstract's "15% improvement" actually matches the 12.8% delta in results/run_2026_04_19.json, then fix flagged claims before the submission deadline. ## Quick Start Run the paper claim audit on my paper directory to verify every number in the tex files matches the raw result files before submission.

Frequently Asked Questions about paper-claim-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify paper numbers match experimental results?▼

Run the audit with the paper directory as argument. It collects all .tex claim files and raw result files (JSON, CSV, YAML configs), then sends them to a fresh cross-model reviewer that traces every number to its evidence and reports match status per claim.

What is a zero-context reviewer in paper auditing?▼

A zero-context reviewer receives only the paper source and raw result files, with no experiment logs, executor summaries, prior audits, or conversation history. This removes confirmation bias so the reviewer compares claims against evidence without expectations about what results should be.

What claim errors does paper claim auditing detect?▼

It detects seven failure modes: number inflation beyond standard rounding, best-seed cherry-picking reported as averages, config mismatches between compared methods, aggregation count mismatches, incorrect delta arithmetic, caption-table mismatches, and scope overclaims like "consistently outperforms" on limited evaluations.

Does the audit block paper submission on failure?▼

No, the audit is advisory and never blocks. A PASS verdict continues normally, WARN flags the draft for number checking, and FAIL prevents marking the paper as submission-ready, but the parent workflow decides whether the verdict blocks finalization.

What happens when a paper has no numeric claims or no raw results?▼

The skill always writes PAPER_CLAIM_AUDIT.json regardless of outcome. A paper with no numeric claims emits verdict NOT_APPLICABLE, while numeric claims without raw result files emit BLOCKED, so downstream verifiers always find the artifact at a predictable path.