evaluation

Evaluate agent performance by scoring outputs against multi-dimensional rubrics covering factual accuracy, completeness, citation accuracy, source quality, and tool efficiency. Run agents against stratified test sets of at least 50 cases and track weighted scores over time.

Build a rubric by defining quality dimensions with descriptive levels from excellent (1.0) to failed (0.0), assigning weights per dimension, and computing a weighted overall score. Set pass thresholds around 0.7 for general use and 0.9 for high-stakes applications.

LLM-as-judge uses a language model to score agent outputs at scale across large test sets. Use a different model family as the judge than the agent being evaluated to avoid self-enhancement bias, and supplement with human review for edge cases.

This usually results from overfitting evals to specific execution paths, test sets that are too small, or missing complexity stratification. Evaluate outcomes rather than steps, use at least 50 test cases, and monitor production traffic continuously with sampling.

Start with 20-30 cases during early development when changes have dramatic impact, then scale to at least 50 cases for reliable signal. Stratify cases across simple, medium, complex, and very complex levels and report scores per stratum.

Yes, sample a fraction of production interactions, evaluate each sample against your rubric, and track pass rate trends. Set alerts when pass rate drops below 0.85 (warning) or 0.70 (critical) to catch gradual degradation.