build-judge

What problem does it solve?

This Skill helps product teams create a focused, repeatable LLM-based evaluator that issues a binary Pass or Fail for one specific failure mode that cannot be reliably checked with code, reducing manual labeling drift and increasing evaluation consistency.

Core Features & Use Cases

• Single-Failure Focus: Build one judge per failure mode (tone mismatch, faithfulness, relevance, completeness, instruction following).
• Few-Shot Grounding: Incorporates PM-provided human-labeled examples including clear Pass, clear Fail, and borderline cases to teach nuance.
• Structured Outputs & Validation: Enforces a critique-first JSON format and requires testing against held-out labels to measure agreement before deployment.

Quick Start

Build a pass/fail LLM judge for the identified failure mode using 20+ labeled pass and fail examples and return a JSON critique and Pass or Fail verdict.