skills.rest
Mark393295827
Mark393295827COMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Evaluate Claude Code sessions with pass/fail criteria and regression tests.

1|Updated Jan 23, 2026
Search Tags:#testing#metrics#evaluation#claude-code#pass@k#eval-driven-development#regression-evals
One-click install
npx skills add https://github.com/Mark393295827/house-maint-ai --skill eval-harness-mark393295827
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/Mark393295827/house-maint-ai/tree/main/skills/eval-harness
Command: npx skills add https://github.com/Mark393295827/house-maint-ai --skill eval-harness-mark393295827

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluating AI-assisted Claude Code tasks is hard without a structured framework; this skill provides a formal evaluation framework to define and run evals, track outcomes, and enforce consistent quality.

Core Features & Use Cases

  • Capability Evals: Define tasks and success criteria to verify new features or behaviors.
  • Regression Evals: Ensure changes do not break existing functionality with baseline checks.
  • Grader Types & Metrics: Supports code-based, model-based, and human graders, with metrics like pass@k and pass^k for reliability measurement.
  • Lifecycle: Define, implement, run, and report evals across development cycles.

Quick Start

Define your evals, implement code to pass them, run checks, and review the resulting eval report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up regression evals for Claude Code sessions?▼

You can evaluate Claude Code tasks by defining capability and regression evals, setting pass/fail criteria, selecting graders, and running checks to generate a structured report with metrics like pass@k.

What metrics are used for eval-driven development in AI testing?▼

Eval-driven development in AI testing uses structured metrics like pass@k and pass^k to measure the reliability and consistency of model capabilities across different versions.

Can I use both human and model-based graders for AI task evaluation?▼

Yes, AI task evaluation supports multiple grader options including code-based, model-based, and human graders to flexibly assess pass/fail criteria across various capability and regression tests.

What is the best way to track capability regressions across model versions?▼

The best way to track capability regressions across model versions is by implementing a formal evaluation framework that defines benchmarks and runs regression evals consistently throughout development cycles.

Do I need dependencies to run pass@k benchmarks on Claude Code?▼

No dependencies are required to run pass@k benchmarks on Claude Code, as the evaluation framework operates independently to define, implement, and report eval outcomes.

Related Skills

Explore More Software Engineering Skills→
yu13130122297yu13130122297

scout

Identify uncertainties in research framing, datasets, metrics, and baselines.

Community
Intermediate
rivaldiekaptrrrrivaldiekaptrrr

eval-harness

Define and run eval suites to validate AI task performance.

Community
Advanced
betaTridentbetaTrident

browser-qa

Automate visual regression testing and UI interaction verification for web applications.

Community
Advanced
ValenBorrasValenBorras
1

eval-harness

Create repeatable evaluation frameworks for Claude Code sessions with pass/fail criteria.

Community
Advanced