skills.rest
agenticgogol
agenticgogolCOMMUNITY
Explore all skills by this author →
GitHub

judge-builder-new

Write binary PASS/FAIL LLM judge prompts grounded in eval specs for routed failure modes.

1|9|Updated Jul 11, 2026
Search Tags:#evaluation#few-shot#model calibration#llm judge#prompt authoring#failure mode#pass fail
One-click install
npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill judge-builder-new-agenticgogol
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: judge-builder-new
Source: https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul/tree/main/.claude/skills/judge-builder-new
Command: npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill judge-builder-new-agenticgogol

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes assets (resource) components.

What problem does it solve? Authoring LLM-as-judge prompts that are actually calibratable is hard: judges drift onto adjacent failure modes, leak dev/test data into few-shot examples, or run under floating model aliases that silently invalidate calibration. This Skill enforces a disciplined authoring process so every judge prompt is binary, spec-grounded, and pinned to an exact model snapshot. ## Core Features & Use Cases - Spec-grounded judge authoring: Builds one judge prompt per failure mode routed to "judge" in evals/evaluators/routing.yaml, citing the exact rule ID from evals/spec.md and the taxonomy.yaml definition. - Five enforced non-negotiables: Binary PASS/FAIL output only, critique-before-label ordering, train-split-only few-shot examples, spec-grounded definitions, and a pinned dated judge model id in the prompt frontmatter. - Guided few-shot curation: Surfaces 4-8 candidate train traces, asks the user to pick 3-6 with confirmed labels, and specifically hunts for a near-miss PASS/FAIL pair that teaches the decision boundary. - Use Case: After evaluator-design-new routes a "hallucinated citation" failure mode to a judge, run this Skill to produce evals/judges/hallucinated_citation.md with a pinned model id, curated train-only examples, and a critique-then-label output contract ready for calibration. ## Quick Start Ask the agent to write a judge prompt for a specific failure mode id that evaluator-design-new routed to the judge method, and confirm the exact dated model snapshot when prompted.

Frequently Asked Questions about judge-builder-new

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I write an LLM judge prompt for a specific failure mode?▼

Run this Skill with the failure_mode_id that evaluator-design-new routed to the judge method. It reads the taxonomy definition and spec.md rule, curates train-split few-shot examples with you, and fills assets/judge_template.md into evals/judges/<failure_mode_id>.md.

Why must an LLM judge output only PASS or FAIL instead of a score?▼

Binary output is required because downstream calibration in judge-align-new computes TPR/TNR over binary confusion matrices. A 1-5 scale, percentage, or confidence score cannot be calibrated by that machinery, so the Skill rejects any non-binary output format.

Can few-shot examples in a judge prompt come from dev or test data?▼

No. Few-shot examples must come only from the train split recorded in evals/labels/<failure_mode_id>_splits.json. Dev and test traces are reserved for calibration, and leaking them into the prompt contaminates the judge's measured performance.

Why does the judge prompt need a pinned dated model id?▼

Calibration results describe one specific model's behavior. A floating alias like gpt-4o can silently resolve to a different snapshot, invalidating the TPR/TNR calibration and the CI regression gate, so the exact dated id is written into the prompt frontmatter.

What happens if no train/dev/test split file exists yet?▼

The Skill stops and tells you to run /judge-split-new for that failure mode first. It will not guess which traces are safe to use as few-shot examples, since using unsplit data risks contaminating later calibration.

Can I build a generic helpfulness or quality judge with this Skill?▼

No. The Skill refuses ungrounded judges and asks which taxonomy failure mode the request maps to. If none exists, it directs you to define a new spec rule or taxonomy code first, because an ungrounded judge cannot be calibrated meaningfully.

Related Skills

Explore More Software Engineering Skills→
adamatdevopsadamatdevops
1

validate-evaluator

Calibrate an LLM judge against human Pass/Fail labels using TPR/TNR metrics.

Community
Advanced
majidraza1228majidraza1228

validate-evaluator

Calibrate an LLM judge against human labels using train/dev/test splits.

Community
Advanced
maragudkmaragudk
10

failure-taxonomy

Builds a structured taxonomy of failure modes from open-coded trace annotations using axial coding.

Official
Advanced
ruban-sruban-s

Evals

Grades AI agent outputs with deterministic assertions and LLM judges across multi-trial eval suites.

Community
Advanced