eval-harness

Create a YAML file under evals/ defining cases with inputs, expected outputs, and graders such as exact_match, contains, regex, or model_based with a rubric. Then write a runner script that executes each case against your system and writes JSONL results.

About 100 cases gives a ±10% Wilson confidence interval, which is the practical floor for quantifying a rate; roughly 385 cases gives ±5%. Ten cases is only a coverage smoke test and cannot support a trustworthy percentage claim.

There is no universal good rate. Hard gates like safety and valid output format require 100% with zero violations, while quality criteria like tone and completeness are tracked as weighted scores where 90-95% is a common convention, not a rule.

Use McNemar's paired test on the same fixed test set, counting cases that flipped from pass to fail versus fail to pass. Comparing two raw percentages is misleading because only flipped cases carry information, and small moves are usually noise.

Use model-based grading for open-ended LLM outputs where exact matching is too rigid, always with an explicit rubric and a score threshold. Calibrate the judge against human labels using Cohen's kappa before trusting its scores to gate anything.

A perfect score usually means the eval is too easy, saturated, or leaked into training data. By the rule of three, zero failures in 20 cases still allows a true failure rate up to about 15%, so add harder and adversarial cases.