What problem does it solve? Eval suites fail for many reasons — weak agent instructions, wrong assertions, judge variance, environment drift — and manually triaging each failure is slow and error-prone. This Skill runs the eval suite, diagnoses every failure against a structured symptom table, applies in-scope fixes, and loops until the suite is green. ## Core Features & Use Cases - Guided eval execution: Run cases by tag (smoke, release, live), by name, or with JSON output carrying judge reasoning, with clear preconditions for Postgres, venv, and API keys. - Structured failure diagnosis: A symptom-to-cause-to-fix table covers judge misses, missing or extra tool calls, flaky runs, HITL gate pauses, teardown cleanup errors, and sweep guard trips. - Scoped repair loop: Fixes are limited to agent instructions, tools, and model config in agents/ and teams/, plus correcting genuinely wrong assertions — never weakening a case to force a pass. - Use Case: An overnight scheduled eval run fails. Use this Skill to locate the failing case in eval history, reproduce it with --name, diagnose the root cause, tighten the agent's routing rule, and re-run until the release tag exits 0. ## Quick Start Run the eval suite, diagnose any failing cases, fix the agent instructions that caused them, and keep iterating until everything passes.
