open-benchmark-methodology

Enforces blind clean-room evaluation methodology for open RTL benchmarks like VerilogEval and CVDP.

26|8|Updated Jun 13, 2026
One-click install
npx skills add https://github.com/vibeic/vibe-ic --skill open-benchmark-methodology-vibeic
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: open-benchmark-methodology
Source: https://github.com/vibeic/vibe-ic/tree/main/vibe-ic-marketplace/plugins/vibe-ic/skills/open-benchmark-methodology
Command: npx skills add https://github.com/vibeic/vibe-ic --skill open-benchmark-methodology-vibeic

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running open RTL benchmarks (VerilogEval, CVDP, RTLLM, PyHDL-Eval) against an AI-driven IC design flow risks overfitting, oracle leakage, and fabricated scores. This Skill enforces a rigorous methodology so every benchmark result is honest, reproducible, and measures the general product path rather than a benchmark-tuned agent. ## Core Features & Use Cases - Single product entry enforcement: All benchmark solving must route through the standard product runner via benchmark_dispatch.py, with thin adapters allowed only for input/output translation. - Blind clean-room evaluation: Forbids reading prior runs, golden outputs, or hidden harnesses; requires fresh full-dataset runs with transcript export and blindness audits. - Program First plus independent AI review: Every candidate passes deterministic gates and a hash-bound independent AI review before official host scoring. - Evidence-backed triage and FLOOR-proof: Failures are classified as product defects, tool gaps, or dataset defects only after running the original golden RTL through the same scorer. - Use Case: When asked to run VerilogEval or CVDP and report a pass@1 score, this Skill governs the entire solve-resume-score pipeline, the per-problem four-stage attribution table, and the mandatory RESULT document with compliance gating. ## Quick Start Run the VerilogEval benchmark cleanly and produce a compliant RESULT.md with the official score, per-problem triage, and clean-room statement.

Frequently Asked Questions about open-benchmark-methodology

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run VerilogEval or CVDP benchmarks with this methodology?▼

Run benchmark_dispatch.py with --solve on a fresh run directory, complete any AI backup or review worklists with --resume until acceptance is COMPLETE, then score with --score using the official host scorer. CVDP additionally requires --scorer-root and the official simulation images.

What is blind clean-room evaluation for RTL benchmarks?▼

It is a fresh full-dataset run that never reads prior results, sibling solutions, hidden harnesses, or golden outputs during solving or review. Transcripts must be exported, and clean-room and blindness audits gate the scoring step.

Can I write benchmark-specific solvers or routing logic?▼

No. Benchmark-specific code may only translate input/output formats and invoke the official scorer. Task classification, RTL authoring, and PASS/FAIL decisions must live in the general product path reachable from ordinary non-benchmark flows.

When can a benchmark failure be declared a dataset defect or floor?▼

Only after the FLOOR-proof: run the official scorer on the candidate, then run the original golden RTL through the same scorer. If the golden also fails and you can quote the failing assertion plus the contradictory prompt lines, it is a genuine floor; otherwise it is your own bug.

Why does a benchmark result get rejected by the compliance gate?▼

The RESULT.md must pass benchmark_result_md_lint.py and the skill compliance check against compliance.yaml, which require a Summary or Verdict section, a Next handoff line, and issue citations for known-systematic tool blocker claims. Missing elements return exit 1 with the gaps listed.