What problem does it solve? Eval results often live only in local runs, so prompt or agent changes can silently regress quality before anyone notices. This Skill wires your eval suite into CI/CD so every pull request is automatically compared against a stored baseline and merges are blocked on real regressions. ## Core Features & Use Cases - Baseline Comparison Script: Generates eval/scripts/compare_to_baseline.py, which reruns the eval via the existing promptfoo config and fails on aggregate score drops or any previously-passing example that now fails. - Judge Trust Gating: Reads eval/calibration_report.md so metrics with judge kappa below 0.6 warn instead of block, keeping uncalibrated judges from halting merges. - PR Comment Reporting: Creates eval/scripts/post_pr_comment.py to post aggregate deltas, regressed example IDs, and per-metric breakdowns directly on the pull request. - Use Case: A team editing prompts in prompts/** wants every PR checked: the generated .github/workflows/eval-gate.yml runs cheap code-based assertions first, then judge-based metrics, and blocks the merge only on calibrated, blocking failures. ## Quick Start Ask the agent to wire the eval into CI by generating a GitHub Actions eval gate that compares each PR against the stored baseline and posts the results as a PR comment.
