skills.rest
DarkCodePE
DarkCodePECOMMUNITY
Explore all skills by this author →
𝕏@natural_select1GitHub

eval-harness

Define, implement, and evaluate AI workflows with pass@k metrics.

Updated Aug 27, 2026
Search Tags:#workflow#testing#metrics#evaluation#benchmarks#ai-development#eval
One-click install
npx skills add https://github.com/DarkCodePE/quipu --skill eval-harness-darkcodepe
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/DarkCodePE/quipu/tree/main/docs/arquetipo/deliverables/skills/_optional/eval-harness
Command: npx skills add https://github.com/DarkCodePE/quipu --skill eval-harness-darkcodepe

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluations are critical for robust AI development. This framework provides a structured approach to define, run, and report evals for Claude Code sessions, enabling consistent quality checks across models and prompts.

Core Features & Use Cases

  • Capability and Regression Evals definitions with reusable templates
  • Multi-grader support: code, model, and human evaluators
  • Metrics collection and reporting (pass@k, pass^k)
  • Integration with the /eval workflow and local storage under .claude/evals
  • Use Case: establish regression tests for agent behaviors and measure impact of updates

Quick Start

Define an eval with /eval, implement it, run /eval check, and generate a final evaluation report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up regression evals for AI agent behaviors?▼

Regression evals for AI agent behaviors are defined using the /eval command, implemented, checked with /eval check, and stored as runnable workflows under .claude/evals to measure the impact of updates.

What metrics are used for tracking AI evaluation reliability?▼

For tracking AI evaluation reliability, the framework collects pass@k and pass^k metrics across projects to enforce consistent quality checks across models and prompts.

What grader types are supported for AI evaluation workflows?▼

Supported grader types for AI evaluation workflows include code, model, and human evaluators, enabling consistent quality checks across different models and prompts.

Do I need external dependencies to run capability evaluations?▼

No external dependencies are required to run capability evaluations. The framework integrates directly with the /eval workflow and stores definitions, implementations, and reports locally under .claude/evals.

When do I need a formal evaluation framework for AI development?▼

A formal evaluation framework for AI development is needed when establishing regression tests for agent behaviors and measuring the impact of prompt or model updates across multiple projects.

Related Skills

Explore More Software Engineering Skills→
yu13130122297yu13130122297

scout

Identify uncertainties in research framing, datasets, metrics, and baselines.

Community
Intermediate
oabdelmaksoudoabdelmaksoud
3

eval-harness

Implement evaluation-driven development with code and model-based evaluators.

Community
Advanced
thanh-abaiithanh-abaii
34

gs:land-and-deploy

Merge pull requests, run CI/CD pipelines, and verify production health.

Community
Advanced
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced