skill-creator

Create, evaluate, and iteratively improve AI agent skills with benchmarked testing.

3|1|Updated Apr 29, 2026
One-click install
npx skills add https://github.com/firstsun-dev/skills --skill skill-creator-firstsun-dev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: skill-creator
Source: https://github.com/firstsun-dev/skills/tree/main/plugins/agent-toolkit/skills/skill-creator
Command: npx skills add https://github.com/firstsun-dev/skills --skill skill-creator-firstsun-dev

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve? Writing effective agent skills is hard to get right on the first try, and there is no systematic way to know whether a skill actually improves agent behavior. This Skill provides a structured draft-test-review-iterate workflow with quantitative benchmarking so skill quality is measured rather than guessed. ## Core Features & Use Cases - Guided Skill Authoring: Interviews the user to capture intent, then drafts a SKILL.md with proper frontmatter, progressive disclosure structure, and writing-style guidance. - Evaluation Harness: Runs test prompts with and without the skill via parallel subagents, grades outputs against assertions, and aggregates pass rates, timing, and token usage into benchmark reports. - Description Optimization: Generates realistic trigger queries, reviews them in an HTML editor, and runs an automated loop that rewrites the skill description to improve triggering accuracy on held-out test queries. - Use Case: You want to build a skill that formats weekly sales reports. This Skill helps you draft it, creates test prompts, runs baseline comparisons, shows results in a browser-based review viewer, and iterates until the outputs pass your review. ## Quick Start Ask the agent to help you create a new skill for your workflow, or to evaluate and improve an existing skill using test cases and benchmarks.

Frequently Asked Questions about skill-creator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a new AI agent skill from scratch?▼

Start by describing what the skill should do and when it should trigger. The skill-creator interviews you about edge cases and output formats, drafts a SKILL.md with proper frontmatter, then runs test prompts to validate the behavior before iterating.

How do I test whether my skill actually improves agent output?▼

Run each test prompt twice in parallel: once with the skill and once without as a baseline. Grade both outputs against assertions, then aggregate pass rates, timing, and token usage into a benchmark.json for comparison.

How can I improve a skill description so it triggers correctly?▼

Generate about 20 realistic should-trigger and should-not-trigger queries, review them in the HTML editor, then run the optimization loop. It rewrites the description and scores each version on train and held-out test queries to avoid overfitting.

Does skill evaluation work without subagents, like on Claude.ai?▼

Yes, but with adaptations. Without subagents you run test prompts sequentially yourself, skip baseline comparisons and quantitative benchmarking, and present outputs inline for qualitative user feedback instead of using the browser viewer.

What is a blind comparison between two skill versions?▼

A blind comparison gives outputs from two skill versions to an independent judge agent labeled only A and B, so the winner is chosen purely on output quality. A post-hoc analyzer then examines transcripts to explain why the winner won.

When should a skill not include test cases?▼

Skills with subjective outputs like writing style or visual design often do not benefit from assertions, since quality requires human judgment. Test cases work best for objectively verifiable outputs such as file transforms, data extraction, or fixed workflow steps.