skills.rest
pjherron
pjherronCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Implements eval-driven development with pass/fail criteria and pass@k metrics for AI coding sessions.

Updated Jul 16, 2026
Search Tags:#llm-as-judge#regression testing#eval-driven development#grader#pass@k#agent evaluation#ai benchmarking
One-click install
npx skills add https://github.com/pjherron/hypoc --skill eval-harness-pjherron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/pjherron/hypoc/tree/main/hypoc/skills/eval-harness
Command: npx skills add https://github.com/pjherron/hypoc --skill eval-harness-pjherron

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? AI-assisted coding lacks objective quality gates, making it hard to know whether an agent's output actually works or whether a prompt change broke existing behavior. This Skill provides a formal evaluation framework that treats evals as unit tests for AI development. ## Core Features & Use Cases - Capability and Regression Evals: Define expected behavior before implementation and verify existing functionality stays intact after changes. - Multiple Grader Types: Use deterministic code graders, model-based LLM-as-judge rubrics, or human review flags depending on the output type. - Reliability Metrics: Track pass@k and pass^k metrics with recommended thresholds (pass@3 >= 0.90 for capability, pass^3 = 1.00 for release-critical paths). - Use Case: Before adding authentication to an app, define capability evals (registration, login, session persistence) and regression evals (public routes, API responses), then run evals after implementation and generate a structured pass/fail report. ## Quick Start Ask the agent to define an eval for a new feature with pass/fail criteria before writing any implementation code.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up eval-driven development for AI coding agents?▼

Define capability and regression evals before writing code, specifying concrete pass/fail criteria for each behavior. Run the evals during and after implementation, then generate a report tracking pass@k metrics to decide if the change is ready to ship.

What is the difference between pass@k and pass^k metrics?▼

pass@k measures whether at least one of k attempts succeeds, indicating practical reliability under retries. pass^k requires all k attempts to succeed, making it a stricter stability test recommended for release-critical regression paths.

When should I use a code grader versus a model grader?▼

Use code graders for deterministic checks like grep patterns, test suites, or build success since they are fast and reproducible. Use model graders for open-ended outputs requiring judgment, and human graders for security-sensitive or ambiguous changes.

What pass rate thresholds should AI agent evals meet?▼

The recommended thresholds are pass@3 of at least 0.90 for capability evals and pass^3 of 1.00 for regression evals on release-critical paths. These balance practical reliability against the cost of retries.

What are common eval anti-patterns to avoid?▼

Avoid overfitting prompts to known eval examples, measuring only happy-path outputs, ignoring cost and latency drift, and allowing flaky graders in release gates. Evals should be versioned with code and kept fast so they actually get run.

Related Skills

Explore More Software Engineering Skills→
mezivillagermezivillager
1

brainstorming

Guides structured design of software features through iterative feedback and approval.

Community
Advanced
kitfunsokitfunso

eval-driven-dev

Define and run capability and regression evals to measure AI workflow quality.

Community
Basic
belokonmbelokonm
8

performance-monitor

Monitor token usage, latency, and eval-driven quality in AI agent workflows.

Community
Advanced
yunusgungoryunusgungor

skillify

Convert raw features into test-covered skills with YAML-frontmattered SKILL.md entries.

Community
Advanced