eval-harness

You can validate Claude Code changes using an eval-driven testing framework that defines capability and regression evaluations, runs multiple graders, and outputs repeatable reports to track baselines.

An eval-driven testing framework formalizes AI code validation through structured eval definitions, deterministic and model-based graders, and tracked metrics like pass@k and pass^k to generate repeatable evaluation reports.

You measure pass@k and pass^k metrics by defining capability evals, running them through code-based or model-based graders, and generating structured evaluation reports across Define, Implement, Evaluate, and Report stages.

Yes, you can use deterministic code-based graders alongside model-based graders to score regression and capability evals, ensuring structured scoring and repeatable evaluation baselines for your changes.

The best way to track evaluation baselines is to use a formal evaluation framework that enforces structured storage of eval definitions and results, comparing new runs against stored baselines in a final report.

A formal evaluation framework solves the problem of unrepeatable validation by codifying expected behavior into regression evals, applying structured graders, and storing baselines to ensure informed decision-making.