What problem does it solve? Reducing LLM eval and agent runtime costs is risky: swapping to a cheaper model, compressing prompts, or routing tasks can silently degrade quality, and savings measured against an uncalibrated judge are meaningless. This Skill enforces a calibrated-judge gate and runs controlled cost experiments so savings never come at the expense of undetected regressions. ## Core Features & Use Cases - Calibration Gate: Refuses to run unless eval/calibration_report.md shows an approved judge (kappa >= 0.6), directing users to judge-calibrator first. - Three Gated Experiments: Model substitution (per-task results against baseline), prompt compression (token savings always paired with score deltas), and a rule-based router tested against existing golden-set labels. - Structured Reporting: Writes eval/cost_report.md with cost/token deltas, aggregate and per-task score deltas, and adopt/reject recommendations per experiment. - Use Case: After running baseline-runner and judge-calibrator on an agent, ask to cut costs — the Skill re-runs the eval with a cheaper model, reports which tasks tolerate it, and recommends selective routing. ## Quick Start Ask the agent to reduce eval costs for the current project, for example by testing a cheaper model against the existing baseline and golden set.
