skill-evals

What problem does it solve?

This Skill provides a framework for building automated evaluation suites to consistently measure and improve the quality of other skills over time.

Core Features & Use Cases

• Automated Evaluation Suites: Define and run tests to assess skill accuracy, output quality, and trigger reliability.
• Code-based Graders: Implement deterministic checks for skills with predictable outputs.
• LLM-as-Judge Graders: Use rubrics for subjective evaluation of open-ended skill responses.
• Continuous Monitoring: Set up pre-deploy validation and ongoing quality assessment.
• Use Case: Before deploying a new version of a customer support skill, use this framework to run automated tests that verify its response accuracy and adherence to predefined quality metrics.

Quick Start

Use the skill-evals skill to set up an automated evaluation for the 'customer-support' skill.