agent-eval

You can compare coding agents by running reproducible benchmarks using declarative task YAML definitions that specify repository commits, prompts, and judge criteria to measure pass rate, cost, time, and consistency.

Run isolated benchmarks using git worktrees and commit pinning to prevent cross-run interference, ensuring that each agent evaluation remains completely reproducible and independent from other test runs.

Evaluating coding agents captures multi-run metrics including pass or fail status, API cost, wall-clock time, and consistency across repeated trials to provide a complete performance comparison.

Yes, agent evaluation supports deterministic judges including pytest, shell commands, grep pattern checks, and LLM-based judges to verify coding agent task completion accurately.

Yes, agent evaluation applies to agent comparison scenarios across local repositories, using task YAML definitions and git worktrees to isolate runs and ensure reproducible results.

Run multiple trials of different agents against representative tasks using reproducible benchmarks, then aggregate multi-run metrics like pass rate and cost to choose agents based on data rather than impressions.