golden-dataset-builder

Define tasks and metrics first, then generate examples in small reviewed batches rather than all at once. This Skill drafts 5-8 examples per batch in a fixed JSONL schema and appends them to eval/golden_set.jsonl only after you confirm each batch.

This Skill targets roughly 50 examples split about 60% common cases, 25% edge cases, and 15% past-failure cases, weighted by task priority. The set is treated as a living artifact that grows over time from production monitoring and failure analysis.

The dataset is written as JSONL to eval/golden_set.jsonl, one object per line. Each record includes a stable sequential ID, task_id, category, input with user message and context, good and bad output notes, applicable metrics, and a source field.

The Skill does not fabricate past-failure examples when no real failure log exists. It relabels that quota slice as edge cases, notes the substitution, and expects the failure slice to grow later from production monitoring.

Both eval/tasks.md and eval/metrics.md must exist before generation starts. If either file is missing, the Skill stops and points you to the prerequisite skill instead of proceeding.