Implements iterative evaluation and refinement loops for improving AI agent outputs.
npx skills add https://github.com/junevm/splathash --skill agentic-eval-junevm
Please help me install this Agent Skill. Skill: agentic-eval Source: https://github.com/junevm/splathash/tree/main/.github/skills/agentic-eval Command: npx skills add https://github.com/junevm/splathash --skill agentic-eval-junevm
High-intent search queries and answers about installing and using this skill.
Use a reflection loop: generate the output, prompt the LLM to evaluate it against your criteria returning structured JSON, then refine based on failed criteria. Repeat up to a maximum of 3 iterations or until all checks pass.
The evaluator-optimizer pattern separates generation and evaluation into distinct components. A generator produces output, an evaluator scores it against dimensions like accuracy and clarity, and an optimizer refines it until the score exceeds a defined threshold.
Prompt the LLM with both outputs and your comparison criteria, asking which is better and why. This works well for ranking candidate generations or A/B testing prompt variations against a defined quality standard.
Set a maximum of 3 to 5 iterations to prevent infinite loops and control costs. Also add convergence detection that stops the loop if the output score stops improving between iterations.
Free-text critiques are unreliable to parse. Instruct the LLM to return structured JSON with explicit PASS/FAIL status per criterion, and handle parse failures gracefully so the loop can recover or exit cleanly.
eric-sabeEvaluate and refine AI agent outputs through iterative self-critique loops.
Veloxia-agencyCompare prompt variants and score outputs for structured LLM tasks.
Nandansai08Builds an eval harness that scores agent runs across prompt, model, and tool changes.
jolequesOrchestrate QA-driven dataset generation workflows for Gemini fine-tuning.
