evals-write-spec

What problem does it solve?

This Skill streamlines the creation and maintenance of evaluation specifications for Large Language Models (LLMs), ensuring robust and reproducible testing.

Core Features & Use Cases

• Spec File Generation: Provides a structured template for defining evaluation suites, including datasets, tasks, and evaluators.
• Dataset & Evaluator Management: Facilitates the definition and integration of custom datasets and various evaluator types (CODE, LLM-as-Judge, RAG, Trace-Based).
• Use Case: When developing a new LLM feature, use this Skill to quickly scaffold the necessary evaluation spec, define test cases with expected outputs, and integrate relevant metrics for performance measurement.

Quick Start

Use the evals-write-spec skill to create a new evaluation spec file for testing LLM responses.