skills.rest
dev-khoi
dev-khoiCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

evaluator-optimizer

Implements a generate-evaluate-revise loop that iterates outputs against a rubric until quality passes.

Updated Apr 29, 2026
Search Tags:#llm evaluation#feedback loop#ai workflow#iterative refinement#evaluator-optimizer#quality rubric#agent pattern
One-click install
npx skills add https://github.com/dev-khoi/AURA-conHack-2026 --skill evaluator-optimizer-dev-khoi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluator-optimizer
Source: https://github.com/dev-khoi/AURA-conHack-2026/tree/main/.opencode/skills/evaluator-optimizer
Command: npx skills add https://github.com/dev-khoi/AURA-conHack-2026 --skill evaluator-optimizer-dev-khoi

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Single-pass AI generation often produces outputs that miss a known quality bar, and there is no built-in mechanism to detect and fix those failures. This Skill provides a structured loop where one component generates a candidate, another scores it against an explicit rubric, and feedback drives revision until the output passes. ## Core Features & Use Cases - Generator-Evaluator Separation: Keeps the generator and the stricter evaluator as distinct prompts so scoring stays objective. - Rubric-Driven Iteration: Defines pass/fail criteria, thresholds, max revision rounds, and plateau stop rules to control cost. - Failure Mode Guidance: Documents common pitfalls such as soft evaluators, vague feedback, and rubric overfitting. - Use Case: When drafting a structured API response or policy-compliant document, generate a draft, score it against schema validity and style rules, then revise with concrete fix hints until it passes the threshold. ## Quick Start Use the evaluator-optimizer pattern to generate a draft, score it against my rubric, and revise it until it passes the quality threshold.

Frequently Asked Questions about evaluator-optimizer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement an evaluator-optimizer loop for AI outputs?▼

Define an explicit pass/fail rubric, then separate the generator and evaluator into distinct prompts. The evaluator returns a score, reasons, and fix hints, and the optimizer revises using that feedback until the output passes or the round limit is reached.

What are good evaluation signals for scoring generated outputs?▼

Strong signals include schema validity, test pass rate, grounding to source facts, style or policy compliance, and ranking scores. Choose signals that can be written or coded objectively rather than vague subjective judgments.

When should I not use an evaluator-optimizer pattern?▼

Avoid it when no reliable evaluation signal exists, feedback is vague or subjective, speed matters more than polish, or revisions drift across a huge output space. High-risk cases should route to human review instead of a model judge.

Why does my evaluation loop keep passing bad outputs?▼

The evaluator is likely too soft or shares blind spots with the generator. Make the evaluator stricter than the generator, give it concrete failure examples, and tune the rubric using saved failing outputs.

How do I stop an evaluator-optimizer loop from running forever?▼

Set a maximum number of revision rounds and a plateau stop rule that halts iteration when scores stop improving. Without explicit stop conditions, cost climbs while quality gains flatten.

Related Skills

Explore More Software Engineering Skills→
rhyme17rhyme17

agentic-eval

Iteratively evaluate and refine AI agent outputs against scoring thresholds.

Community
Intermediate
ibytechaosibytechaos
1

santa-method

Verifies generated output through dual independent review agents with a convergence loop.

Community
Advanced
gozonerdgozonerd

Comparative Ranking Methodology

Generates prompt variants, scores outputs against quality criteria, and selects the strongest result.

Community
Intermediate
VictoriakaeyVictoriakaey

critic-judge-design

Design Critic/Judge nodes with input schemas and structured verdicts.

Community
Advanced