What problem does it solve? Agent changes ship blind: every prompt tweak, model upgrade, or tool swap is a bet with no scoreboard, and nondeterministic runs make single-run spot checks meaningless. This Skill builds a repeatable eval harness so agent changes ship with measured evidence instead of vibes. ## Core Features & Use Cases - Task Set Construction: Harvests real production tasks, pinned past failures, and synthetic edge cases into a version-controlled suite with success criteria and category tags. - Variance-Aware Scoring: Runs each task multiple times, reports pass rates with run counts, and establishes per-task variance bands so noise is never mistaken for regression. - Judge Calibration & Regression Gating: Calibrates LLM judges against human labels, freezes baselines with config fingerprints, and gates changes on per-category deltas wired into CI. - Use Case: Migrating a ticket-triage agent to a new model, the suite reveals the aggregate score improved while the hardest ticket category regressed 23 points — invisible without category-level readouts. ## Quick Start Build an eval harness for my agent that runs a versioned task set multiple times, scores results with checkable criteria and a calibrated judge, and gates prompt or model changes on per-category regression deltas.
