754,000+ skills indexed

CxxxxDxxxFCOMMUNITY

Explore all skills by this author →

Verified:GitHub

evaluation

Create multi-dimensional rubrics and stratified test sets for AI agent evaluation.

Updated Feb 26, 2026

Search Tags:#quality assurance #llm-as-judge #evaluation #performance metrics #rubrics #agent testing

One-click install

npx skills add https://github.com/CxxxxDxxxF/project-blackout --skill evaluation-cxxxxdxxxf

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: evaluation
Source: https://github.com/CxxxxDxxxF/project-blackout/tree/main/Agent-Skills-for-Context-Engineering/skills/evaluation
Command: npx skills add https://github.com/CxxxxDxxxF/project-blackout --skill evaluation-cxxxxdxxxf

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a framework for systematically evaluating the performance, quality, and effectiveness of AI agent systems, enabling continuous improvement and validation of context engineering choices.

Core Features & Use Cases

Multi-Dimensional Rubrics: Define and apply rubrics covering factual accuracy, completeness, citation accuracy, source quality, and tool efficiency.
LLM-as-Judge & Human Evaluation: Supports scalable automated evaluation and crucial human review for edge cases.
Test Set Design: Tools for creating and managing test sets stratified by complexity.
Continuous Evaluation: Enables building pipelines for automated testing and production monitoring.
Use Case: A team developing a research agent can use this skill to create a test set of complex queries, run the agent against it, and analyze the results using a multi-dimensional rubric to identify areas for improvement before deployment.

Quick Start

Use the evaluation skill to build a test framework for agent performance.

Frequently Asked Questions about evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate AI agent performance for non-deterministic outputs?▼

To evaluate non-deterministic AI agent performance, apply multi-dimensional rubrics covering factual accuracy and tool efficiency, design stratified test sets, and use LLM-as-judge methodologies to measure outcomes systematically.

What is the best way to create a test set for agent testing?▼

The best way to create a test set for agent testing is to stratify queries by complexity, enabling systematic assessment of your AI agent across varying difficulty levels before deployment.

How does LLM-as-judge work for quality assurance of AI agents?▼

LLM-as-judge works for quality assurance by providing a scalable automated evaluation methodology to score AI agent responses against multi-dimensional rubrics, reducing the need for manual human review on standard cases.

Can I use human evaluation alongside LLM-as-judge for agent testing?▼

Yes, you can use human evaluation alongside LLM-as-judge to handle edge cases, ensuring crucial review where automated evaluation may fail to capture nuanced context or factual accuracy issues.

How do I build an automated testing pipeline for continuous agent evaluation?▼

Build a continuous evaluation pipeline by integrating stratified test sets and multi-dimensional rubrics into your automated testing workflows, enabling production monitoring and validating context engineering choices.

What dimensions should an AI agent evaluation rubric cover?▼

An AI agent evaluation rubric should cover factual accuracy, completeness, citation accuracy, source quality, and tool efficiency to provide a comprehensive framework for measuring agent effectiveness.

Related Skills

Explore More Software Engineering Skills→

debug-issue

Diagnose bugs by tracing evidence and generating a BR issue template.

evaluation

Construct evaluation suites with multi-dimensional rubrics and automated scoring for agent systems.

agent-evaluation

Develop evaluation frameworks for non-deterministic agent systems.

evaluation

Evaluate agent outputs against multi-dimensional rubrics to compute scores and detect regressions.