What problem does it solve? AI-assisted coding lacks objective quality gates, making it hard to know whether an agent's output actually works or whether a prompt change broke existing behavior. This Skill provides a formal evaluation framework that treats evals as unit tests for AI development. ## Core Features & Use Cases - Capability and Regression Evals: Define pass/fail criteria before implementation and verify that changes do not break existing functionality. - Multiple Grader Types: Use deterministic code graders, model-based LLM-as-judge graders, or human review flags depending on the output type. - Reliability Metrics: Track pass@k and pass^k metrics to measure agent reliability across attempts and model versions. - Use Case: Before adding an authentication feature, define capability evals (user registration, login, session persistence) and regression evals (existing routes unchanged), then run evals after implementation and generate a report showing pass@3 of 100% before shipping. ## Quick Start Ask the agent to define an eval for a new feature with pass/fail criteria, then implement the feature and run the evals to generate a pass/fail report.
