agentic-eval

Evaluate and refine agent outputs using rubrics, reflection loops, and evaluator-optimizer pipelines.

1|Updated Apr 3, 2026
One-click install
npx skills add https://github.com/yiwei79/root-azoth --skill agentic-eval-yiwei79
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agentic-eval
Source: https://github.com/yiwei79/root-azoth/tree/main/.opencode/skills/agentic-eval
Command: npx skills add https://github.com/yiwei79/root-azoth --skill agentic-eval-yiwei79

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Single-shot generation often produces outputs that miss quality bars for code, reports, and analysis. This Skill provides structured patterns for iterative self-evaluation and refinement so agent outputs converge on defined criteria before delivery. ## Core Features & Use Cases - Reflection loops: Generate, critique against criteria, and revise outputs for up to a bounded number of iterations with structured JSON feedback. - Evaluator-optimizer pipelines: Separate generation and scoring components with configurable thresholds (default 0.85) and weighted rubric dimensions like accuracy, clarity, and completeness. - Code-specific refinement: Test-driven loops that generate tests, run them, and revise code until the suite passes. - L2 evidence records: Normalize evaluator and reviewer output into JSON evidence records appended via the governed append script for downstream prompt-engineering refinement. - Use Case: A pipeline stage produces a design document; the evaluator scores it against the brief and governance rules, and the optimizer revises it until the score clears the threshold before the next stage proceeds. ## Quick Start Ask the agent to evaluate the current draft against a defined rubric and iteratively refine it until the overall score reaches 0.85 or the iteration limit is hit.

Frequently Asked Questions about agentic-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I make an LLM evaluate and improve its own output?▼

Use a reflection loop: generate an output, evaluate it against explicit criteria with structured JSON critique, then revise only the failing dimensions. Repeat up to 3-5 iterations and stop early when all criteria pass or scores stop improving.

What is the evaluator-optimizer pattern in agent workflows?▼

The evaluator-optimizer pattern separates generation and scoring into distinct components. A generator produces a candidate, an evaluator returns structured dimension scores, and an optimizer revises the output until the overall score crosses a threshold such as 0.85.

How do I evaluate LLM outputs with a rubric?▼

Define weighted dimensions such as accuracy (0.4), clarity (0.3), and completeness (0.3), then have a judge score each dimension from 1-5. Compute the weighted total divided by 5 to get a normalized score comparable across outputs.

When should I not use iterative evaluation loops?▼

Avoid reflection loops for one-off formatting, trivial copy edits, or tasks without clear success criteria, since the extra generation cycles add cost without measurable quality gain. Always set iteration limits and convergence checks to prevent infinite loops.

Why does my evaluation loop never converge?▼

Non-convergence usually comes from vague criteria, an unreachable threshold, or a judge that scores inconsistently. Tighten the rubric, lower or justify the threshold, log every iteration's scores, and stop when improvement between iterations flatlines.