What problem does it solve? Skill behavior claims are hard to verify: descriptions drift from real user phrasings, trigger accuracy is guessed from prose, and there is no durable record of whether a skill actually works. This Skill runs structured eval cases through the real Claude Code CLI in isolated workspaces and accounts for every result in an append-only, fully-attributed JSONL ledger. ## Core Features & Use Cases - run: Executes trigger, execution, and retirement-tier cases from a skill's evals/eval-cases.yaml with denied-by-default mutating tools, 3-trial defaults, per-case fresh contexts, and a --max-runs cost cap; timeouts and missing evidence record as error, never as passes. - report: Computes pass rates, capability-to-regression graduation, saturation and retirement signals per skill x tier x model, plus --paired with-skill vs skill-masked output uplift and --efficiency duration/token reads. - sync and score: Deterministically detects drift between eval sets and the real-phrasing corpus (exit 1 on drift, clean pass on fresh hosts), and appends human scorecard verdicts for subjective hitl cases under the judge-verdict schema. - Use Case: Before installing or exporting a skill, run the shared adversarial pack (--skill _shared) to probe prompt-injection, scope-bypass, and ban-term leakage as an explicit pre-install gate. ## Quick Start Ask the assistant to run the evals for a specific skill with an explicit model, for example: run the evals for transcript-fetcher with one trial and then report its pass rate.
