skill-evaluation

Evaluate an agent skill by running candidates on the same realistic tasks with identical inputs and constraints, then judging the produced artifacts against a short shared rubric. Hide candidate identities before review so you judge the work, not the skill's claims.

Test triggering separately by writing a small set of realistic prompts, including some that should trigger the skill and some that should not. For each prompt, record whether it should trigger, whether it did, and why the result was correct or incorrect.

A good rubric is short, task-specific, and uses observable criteria such as factual correctness, scope control, and verification evidence. Avoid rubrics that reward a candidate for repeating the skill's own instructions or rely on word counts for subjective work.

Skip evaluation for small wording changes unless the change carries meaningful uncertainty or risk. Reserve full evaluations for new public skills, meaningful changes to existing skills, or skills that trigger at the wrong time or produce poor work.

Blinding prevents bias toward the new or preferred skill when judging artifacts. Reading the actual work and its verification evidence, rather than an agent's summary of intent, keeps the comparison grounded in observable outcomes.