What problem does it solve? AI features that ship on vibes have no held-out dataset, no pass threshold, and no way to detect quality regressions until users complain. This Skill designs a repeatable, dataset-backed evaluation plan for any model-backed product feature before and after it ships. ## Core Features & Use Cases - Measurable Success Criteria: Converts vague goals like "good answers" into concrete targets such as extraction F1 >= 0.9, refusal rate <= 1%, or p95 latency bounds. - Eval Dataset Specification: Defines the held-out labeled set including size, sourcing (production samples, synthetic, adversarial), label schema, and a split with no train/eval leakage. - Tiered Grading Strategy: Assigns code-based grading (exact match, F1, regex) to objective criteria and LLM-as-judge with a locked rubric only to nuanced criteria, then sets a suite-level pass threshold and regression rule persisted as AI_EVAL_PLAN.md. - Use Case: Before shipping an LLM-powered summarization feature, use this Skill to produce an eval plan with a 200-case gold set, per-criterion grading tiers, and a pass threshold wired into the slice's exit criteria. ## Quick Start Ask the AI to design an evaluation plan for your model-backed feature, defining measurable success criteria, a held-out labeled dataset, grading methods, and a pass threshold saved as AI_EVAL_PLAN.md.
