evaluator-design-new

What problem does it solve? After axial coding produces a taxonomy of agent failure modes, teams must decide which failures can be caught by deterministic code checks and which require an LLM judge, then actually implement the code checks. This Skill makes that routing decision systematically and writes the pytest-style evaluators. ## Core Features & Use Cases - Routing Decisions: Classifies each failure mode in evals/taxonomy.yaml as code or judge using documented criteria, recording a one-line rationale per decision in evals/evaluators/routing.yaml. - Code Eval Generation: Writes pytest-style check functions that take a Trace and return a boolean, validated against known-failing and known-passing traces from evals/traces.jsonl. - Mix Enforcement: Targets roughly 2-3 code evals per 1-2 judges and flags backlogs that skew too heavily toward judges. - Use Case: After running axial coding on agent traces, use this Skill to decide that "skipped eligibility check" is a tool-order code check while "tone mismatch" needs a judge, then generate the code eval for the former. ## Quick Start Route the failure modes in evals/taxonomy.yaml to code checks or judges and write the code evaluators.