skills.rest
sori883
sori883COMMUNITY
Explore all skills by this author →
Website𝕏@sori883GitHub

eval-harness

Evaluate Claude Code sessions with pass@k metrics and structured graders.

Updated Jan 15, 2026
Search Tags:#testing#qa#claude#evaluation#framework#code#eval
One-click install
npx skills add https://github.com/sori883/certification-proflie --skill eval-harness-sori883
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/sori883/certification-proflie/tree/main/.claude/skills/eval-harness
Command: npx skills add https://github.com/sori883/certification-proflie --skill eval-harness-sori883

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Claude Code sessions benefit from a formal evaluation framework that defines expected behavior before coding, enables eval-driven development, and tracks regressions with pass@k metrics.

Core Features & Use Cases

  • Capability Eval and Regression Eval to validate new features and prevent regressions in Claude Code workflows.
  • Multiple graders: code-based, model-based, and human reviews to cover deterministic and qualitative assessments.
  • Metrics & Workflow: supports pass@k and pass^k metrics, plus an end-to-end eval workflow from definition to reporting.

Quick Start

Define your eval plan and run the evaluation workflow to produce a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up an evaluation framework for Claude Code sessions?▼

To set up a Claude Code evaluation framework, define your eval plan using capability and regression eval definitions, then run the structured workflow to generate automated reports.

What is eval-driven development and how does it track regressions?▼

Eval-driven development defines expected behavior before coding to prevent regressions. It tracks capability changes over time using pass@k metrics and deterministic graders.

Can I use both automated and human reviews to grade Claude Code evaluations?▼

Yes, Claude Code evaluations support multiple graders including code-based deterministic checks, model-based reviews, and human reviews to cover both quantitative and qualitative assessments.

What is the best way to measure pass@k metrics in code evaluation workflows?▼

The best way to measure pass@k metrics in evaluation workflows is using a formal harness that runs capability and regression tests, applying deterministic and model graders to produce automated reports.

Does this evaluation framework support pass^k metrics alongside pass@k?▼

Yes, the evaluation framework supports both pass@k and pass^k metrics to provide comprehensive evaluation coverage for Claude Code capability and regression testing.

When do I need regression evals for Claude Code workflows?▼

You need regression evals when validating new features in Claude Code workflows and want to prevent existing capabilities from degrading, tracking these changes with pass@k metrics.

Related Skills

Explore More Software Engineering Skills→
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
oabdelmaksoudoabdelmaksoud
3

eval-harness

Implement evaluation-driven development with code and model-based evaluators.

Community
Advanced
zhmxiaowozhmxiaowo
2

eval-harness

Define automated evaluation criteria for Claude Code tasks with multiple graders.

Community
Advanced
nazrulsoftwaredevnazrulsoftwaredev

eval-harness

Define and run evaluation suites for AI-assisted development sessions.

Community
Advanced