experiment-audit

Audit experiment integrity using cross-model review to detect fake ground truth and phantom results.

Updated Aug 18, 2026
One-click install
npx skills add https://github.com/Nitrogen216/awesome-dsh-mods --skill experiment-audit-nitrogen216
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-audit
Source: https://github.com/Nitrogen216/awesome-dsh-mods/tree/main/modes/dsh-autoresearch/vendor/aris-upstream/skills/experiment-audit
Command: npx skills add https://github.com/Nitrogen216/awesome-dsh-mods --skill experiment-audit-nitrogen216

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? LLM agents can unintentionally produce fraudulent experimental results, such as fake ground truth derived from model outputs, self-normalized scores, phantom results referencing nonexistent files, and overstated evaluation scope. This Skill adds an independent integrity constraint by having an external reviewer model audit the experiment before claims are written. ## Core Features & Use Cases - Cross-Model Integrity Review: The executor only collects file paths while an external reviewer backend (Codex MCP or Manual Review MCP) reads the code and judges integrity, preserving reviewer independence. - Six-Point Audit Checklist: Checks ground truth provenance, score normalization, result file existence, dead code detection, scope assessment, and evaluation type classification, each with PASS/WARN/FAIL verdicts and file:line evidence. - Structured Reports: Writes EXPERIMENT_AUDIT.md and EXPERIMENT_AUDIT.json with an overall verdict, per-check details, action items, and claim impact tags consumed by downstream pipeline skills. - Use Case: After an autonomous research pipeline finishes experiments, run the audit to verify that reported metrics come from real dataset ground truth and that every claimed number matches an actual result file before writing the paper. ## Quick Start Ask the agent to audit the experiment results in your project directory for integrity before writing any claims.

Frequently Asked Questions about experiment-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit LLM experiment results for integrity?▼

Run the audit after experiments complete and before writing claims. The executor collects paths to eval scripts, result files, trackers, and paper claims, then an external reviewer model reads everything and returns PASS, WARN, or FAIL verdicts per check.

What fraud patterns does experiment integrity checking detect?▼

It detects fake ground truth derived from model outputs, score normalization using the model's own statistics, phantom results referencing nonexistent files or mismatched numbers, dead metric code never called, and scope language exceeding actual evidence.

Which reviewer backends does the audit support?▼

It supports Codex MCP as the default reviewer and Manual Review MCP as an alternative, selected via a reviewer parameter. Manual review requires a Reviewer-Model identity header; if unavailable, the audit emits REVIEW_UNAVAILABLE rather than falling back silently.

Does the experiment audit block the research pipeline on failure?▼

No, the audit is advisory and never blocks the pipeline. On WARN or FAIL it prints alerts and tags downstream claims with integrity status so reviewers and paper-writing steps can see the concerns.

Why must the reviewer be a different model family than the executor?▼

Cross-model review prevents the executor from acquitting its own work, since same-family models share failure modes. The executor only collects file paths and never participates in integrity judgment, preserving reviewer independence.