eval-testing

What problem does it solve?

This Skill provides a robust framework for developing, running, and analyzing evaluations of agent behavior, ensuring consistency and quality across different prompts, tools, models, and configurations.

Core Features & Use Cases

• YAML Eval Case Creation: Define detailed test cases with inputs, mocks, and expected outcomes.
• Assertion Types: Verify tool selection, response quality, skill invocation, and multi-step workflows.
• Multi-Model Testing: Run evaluations across a matrix of different LLM models.
• LLM-as-Judge Scoring: Use an LLM to score response quality against defined criteria.
• Use Case: When developing a new agent feature, use this Skill to write comprehensive tests that confirm the agent correctly identifies and uses the appropriate tools for user requests, and that its responses meet quality standards.

Quick Start

Run all agent behavior evaluations using the default model.