skills.rest
tanveerriaz
tanveerriazCOMMUNITY
Explore all skills by this author →
GitHub

llm-eval-golden-set

Builds a golden set of test cases and an eval runner to detect LLM output regressions.

1|Updated Jun 3, 2026
Search Tags:#llm-as-judge#llm evaluation#regression testing#prompt testing#quality baseline#eval runner#golden set
One-click install
npx skills add https://github.com/tanveerriaz/Skillz --skill llm-eval-golden-set-tanveerriaz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-eval-golden-set
Source: https://github.com/tanveerriaz/Skillz/tree/main/skills/llm-eval-golden-set
Command: npx skills add https://github.com/tanveerriaz/Skillz --skill llm-eval-golden-set-tanveerriaz

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Prompt tweaks and model swaps silently change LLM feature behavior, and manual spot-checks on one or two inputs fail to catch regressions before they reach production. ## Core Features & Use Cases - Golden Set Creation: Curate 10-30 representative cases covering common inputs, edge cases, and past failures, stored as versioned JSON data. - Repeatable Eval Runner: Build a script that runs each case against the feature and scores output against explicit expectations, using string checks for objective criteria or LLM-as-judge for subjective ones. - Baseline Tracking: Record a baseline score and re-run evals on every prompt or model change to catch quality drops before release. - Use Case: After changing a summarization prompt, re-run the eval suite and see the score drop from 19/20 to 14/20, revealing a regression before it ships. ## Quick Start Create a golden set and eval runner for my LLM summarization feature so I can catch regressions when I change prompts or models.

Frequently Asked Questions about llm-eval-golden-set

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test an LLM feature for regressions after changing a prompt?▼

Create a golden set of 10-30 representative inputs with explicit expected qualities, then run an eval script that scores each output against those expectations. Compare the score to your recorded baseline after every prompt or model change.

What is a golden set in LLM evaluation?▼

A golden set is a curated collection of representative inputs paired with expected output qualities, stored as versioned data like JSON. It covers common cases, edge cases, and past failures so regressions can be detected repeatably.

How do I evaluate subjective LLM output quality?▼

Use a rubric or LLM-as-judge approach with a fixed grading prompt kept consistent across runs. For objective criteria like format, length, or required content, use deterministic string and structure checks instead.

How do I handle non-deterministic LLM outputs in evals?▼

Run multiple samples per case or assert on stable properties rather than exact string matches. Check qualities like required content, format, and length bounds instead of expecting identical output text.

What are the limitations of a small golden set eval?▼

A small set may miss rare failure modes and can be gamed if prompts are tuned to memorize cases. Keep cases representative, grow the set with every new real-world failure, and never include real personal data or secrets.

Related Skills

Explore More Software Engineering Skills→
iani-kuliiani-kuli

generate-synthetic-data

Generate diverse synthetic test tuples for LLM pipeline evaluation.

Community
Intermediate
AraflyArafly

cicd-pipeline-review

Audit CI/CD pipeline estates for triggers, runners, secrets, and deployment paths.

Community
Advanced
CAPHTECHCAPHTECH
1

llm-eval-designer

Designs LLM evaluation plans with failure-mode coverage and test-case templates.

Official
Advanced
AltairaLabsAltairaLabs
9

codegen-metrics

Emit structured JSON metrics from codegen evaluation scripts.

Official
Basic