skills.rest
chrissstellee
chrissstelleeCOMMUNITY
Explore all skills by this author →
GitHub

bench-skills

Benchmarks skill quality on held-out eval prompts with reproducible LLM-judge scoring.

Updated Jul 26, 2026
Search Tags:#optimization#benchmark#scoring#skill evaluation#reproducible#llm judge#eval prompts
One-click install
npx skills add https://github.com/chrissstellee/Movix --skill bench-skills-chrissstellee
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bench-skills
Source: https://github.com/chrissstellee/Movix/tree/main/.agents/skills/bench-skills
Command: npx skills add https://github.com/chrissstellee/Movix --skill bench-skills-chrissstellee

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? After optimizing a skill, there is no objective way to know whether the changes actually improved it. This Skill measures skill quality on held-out eval prompts with a strict LLM judge, so you can quantify the lift from optimization instead of guessing. ## Core Features & Use Cases - Held-out eval sets: Builds representative prompt/rubric pairs from saved bench files, mined trace spans, or synthesized prompts, sampled deterministically with fixed seeds. - Reproducible scoring: Judges each skill's SKILL.md as instructions on a 1-5 scale against rubrics, with identical scores for identical skill text and seeds. - Before/after comparison: Scores the current skill against its most recent backup to report the delta from optimization, flagging regressions and suggesting a restore. - Use Case: After running an optimization pass on your skills, ask to bench them with seed 42 and receive per-skill mean scores, per-prompt breakdowns, and an overall lift report. ## Quick Start Ask the assistant to bench your skills with max 5 samples and seed 42 to get a scored quality report.

Frequently Asked Questions about bench-skills

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure whether skill optimization actually helped?▼

Run the bench flow in before/after mode: it scores the current installed skill and the most recent backup using the same prompts and seeds, then reports the delta. A positive lift means the optimization improved expected response quality.

How do I benchmark a single skill instead of all skills?▼

Pass the skill name as an argument, for example by asking to bench just that skill. You can also cap eval prompts with --max-samples and fix reproducibility with --seeds.

Where do the benchmark eval prompts come from?▼

Prompts come from a saved bench file at ~/.stellar/bench/<skill>.jsonl if present, otherwise from real opening prompts mined from trace spans, or from synthesized prompts based on the skill description when no data exists.

Are benchmark results reproducible across runs?▼

Yes. Sampling and judging use fixed seeds, so the same skill text with the same seed yields the same score. Reuse the same seed across before/after runs for apples-to-apples comparison.

Does benchmarking modify my skills?▼

No. The bench flow is strictly read-only: it never edits skills, only produces scores and a report. If a regression is detected, it suggests restoring from a backup as a separate step.

What are the limitations of skill benchmarking with small eval sets?▼

Synthetic or tiny eval sets are a weak signal and the report says so explicitly. Scores reflect expected response quality judged from the skill text, not live execution results.

Related Skills

Explore More Software Engineering Skills→
stellar-zkstellar-zk
3

optimize-skills

Rewrites installed skill files using mined examples from local usage traces.

Official
Advanced
bfowlebfowle

backup-and-restore

Back up and restore dotfiles in Unix-like environments with timestamped manifests.

Community
Basic
avsmavsm
35

memtrace

Instrument OCaml builds with memtrace to collect and compare allocation traces.

Community
Advanced
applied-artificial-intelligenceapplied-artificial-intelligence
81

llm-evaluation

Evaluate LLM applications with automated metrics and LLM-as-judge scoring.

Official
Advanced