Evals

Write a YAML suite file with a name, type (capability or regression), pass_threshold, agent_level, judge_level, trials, and a cases list where each case has an id, prompt, and assert array. Run it with the EvalRunner tool using the suite name flag.

pass@k reports whether any trial passed, while pass^k requires every trial to pass, making it the honest metric for reliability-critical behavior. A flaky case passing two of three trials scores pass@k of 1 but pass^k of 0.

Use deterministic asserts (contains, regex, is-json, length bounds) for any mechanical property since they are free and exact. Reserve llm-rubric and llm-assert for nuance only a reader can catch, targeting roughly 60/40 deterministic-to-model weight.

A judge sharing the subject's inference rung flatters it, degrading discrimination between good and bad outputs. The default pairs a medium subject with a high judge, and close calls benefit from multi-judge panels.

Yes, the ComparePrompts workflow runs each variant against the same suite with position swapping to counter presentation bias, then reports p-values and confidence intervals. Pre-commit your success criterion and falsifier before running.

Single-shot subjects have no tools, so agentic system prompts make them narrate tool calls instead of answering; EvalRunner appends an evaluation-context suffix to counter this. Multi-turn behavior requires the separate scenario runner with an Anthropic API key.