Defines and runs eval-driven development criteria with pass/fail graders and pass@k metrics.
npx skills add https://github.com/inuishan/PET --skill eval-harness-inuishan
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/inuishan/PET/tree/main/.codex/skills/eval-harness Command: npx skills add https://github.com/inuishan/PET --skill eval-harness-inuishan
/eval check add-authentication during implementation and generate a final report showing 5/5 capability evals passed with pass@3 at 100%. ## Quick Start Ask the agent to define an eval-driven development plan with pass/fail criteria for the feature you are about to implement.High-intent search queries and answers about installing and using this skill.
Define capability and regression evals before writing code, specifying success criteria for each task. Run evals during implementation with code-based graders like grep or npm test, then generate a report tracking pass@k metrics.
pass@k measures whether at least one of k attempts succeeds, with pass@3 above 90% as a typical target. pass^k requires all k trials to succeed, a stricter bar used for critical paths and regression evals.
Record a baseline SHA or checkpoint, list existing tests with PASS/FAIL status, and compare results against the previous run. A regression eval reports X/Y passed versus the baseline to detect broken functionality.
Use code-based graders for deterministic checks like build success or test passes. Use model-based graders for open-ended outputs requiring judgment on structure, edge cases, and error handling, scored on a 1-5 scale.
No, security-sensitive changes should be flagged for human review with a risk level of LOW, MEDIUM, or HIGH. The framework explicitly recommends never fully automating security checks.
The-Skyy-Rose-Collection-LLCDefine, run, and report evaluations with /eval commands.
pjherronImplements eval-driven development with pass/fail criteria and pass@k metrics for AI coding sessions.
s3nex-comGuide LLM feature development with eval harnesses, prompt versioning, and deployment gates.
rabbicseWrite unit tests with Jest, Vitest, or pytest using Arrange-Act-Assert and coverage gating.