evaluation

What problem does it solve?

This Skill addresses the problem of inconsistent, subjective, or incomplete evaluation of AI features, which leads to unreliable performance and difficulty in tracking improvements. It standardizes the creation of comprehensive evaluation suites, ensuring your AI models are rigorously tested and validated.

Core Features & Use Cases

• Standardized Eval Structure: Provides templates for spec.md and rubric.md to ensure consistent and clear evaluation design across all features.
• Mixed Validation Types: Guides you in using both code-based (deterministic checks) and LLM-as-judge (quality assessment) validations for comprehensive coverage.
• Objective Rubric Creation: Emphasizes writing concrete, objectively verifiable criteria for LLM-based evaluations, reducing subjectivity and improving reliability.
• Use Case: When developing a new AI feature, use this Skill to create a robust evaluation suite that includes a detailed specification of what to test, a clear rubric for LLM-as-judge assessments, and a plan for both code-based and LLM-based validations, ensuring high-quality AI outputs and faster iteration.

Quick Start

Help me create a new evaluation suite for an AI feature that generates code, starting with the spec.md template and defining a few code-based and LLM-judged requirements.