What problem does it solve? Changing a skill and trusting it based on a single eyeballed run leads to false conclusions, because agent runs are non-deterministic and self-judged output is biased. This Skill provides a rigorous playbook for measuring whether a skill change actually helps, using controlled A/B comparisons with a calibrated noise floor. ## Core Features & Use Cases - Harness selection by skill type: Routes conversational skills to NTM-driven tmux replay harnesses, single-shot transforms to headless claude -p two-arm runs, and description quality to trigger evals. - Fixture authoring from real history: Pins real tasks to immutable start SHAs with mined or requirements-derived prompts and human turns, avoiding contaminated checkouts. - Eval validation before trusting results: Measures the noise floor (same arm, same fixture, ≥3 runs), confirms a quality floor, and proves generality on a second fixture before reading any A/B. - Use Case: Before merging a v3 planning skill, replay two real historical tasks against v2 and v3 with three trials per arm, then compare median tokens and quality to confirm the new version is cheaper at equal quality. ## Quick Start Ask the agent to set up an A/B eval comparing the current and proposed versions of a skill using real tasks replayed from repo history.
