What problem does it solve? AI-assisted coding lacks a systematic way to verify that agent outputs actually meet requirements and that new changes do not break existing behavior. This Skill provides a formal evaluation framework that treats evals as the unit tests of AI development, letting you define pass/fail criteria before implementation and measure agent reliability over time. ## Core Features & Use Cases - Capability and Regression Evals: Define structured eval templates that test whether Claude can accomplish new tasks and whether existing functionality still works after changes. - Multiple Grader Types: Use deterministic code-based graders (grep checks, test runs, build verification), model-based graders for open-ended outputs, or human review flags for high-risk changes. - Reliability Metrics: Track pass@k (at least one success in k attempts) and pass^k (all k trials succeed) to quantify agent reliability across model versions and prompt changes. - Use Case: Before adding authentication to your app, define capability evals (user registration, login, session persistence) and regression evals (public routes still work), implement the feature, then run the evals and generate a report showing pass@3 of 100% before shipping. ## Quick Start Ask the AI to define an eval-driven development plan with capability and regression evals for the feature you are about to build.
