What problem does it solve? It answers whether an agent skill actually improves output quality by running controlled with-skill versus without-skill comparisons, grading results against evidence-based assertions, and aggregating pass rates, timing, and token costs into benchmarks. ## Core Features & Use Cases - Structured Eval Loop: Design test cases in evals.json, run each prompt with and without the skill in clean contexts, grade outputs, and iterate across versioned iteration directories. - Static Pre-Eval Analysis: Score executability, invocability, collision risk against neighbor skills, over-specification risk, and taxonomy fit before running any evals. - Programmatic Verification and Benchmarking: Use verify.py for mechanical assertion checks (file existence, valid JSON, counts) and benchmark.py to compute pass-rate, time, and token deltas. - Use Case: After writing a new CSV-analysis skill, run three test prompts with and without the skill, grade the outputs, and read benchmark.json to see the skill raised pass rate by 50 points at a known token cost. ## Quick Start Ask the agent to evaluate the skill in a given directory by designing test cases, running with-skill and baseline comparisons, and producing a benchmark report.
