skills.rest
Onlyaguest
OnlyaguestCOMMUNITY
Explore all skills by this author →
GitHub

prompt-tuner

Validate prompt outputs against a golden dataset with OpenAI or Gemini.

18|2|Updated Mar 18, 2026
Search Tags:#testing#versioning#llm#regression#tuning#prompt#golden-dataset
One-click install
npx skills add https://github.com/Onlyaguest/ViviStableSkills --skill prompt-tuner-onlyaguest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: prompt-tuner
Source: https://github.com/Onlyaguest/ViviStableSkills/tree/main/prompt-tuner
Command: npx skills add https://github.com/Onlyaguest/ViviStableSkills --skill prompt-tuner-onlyaguest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, python-dotenv, pyyaml.

What problem does it solve?

This skill automates prompt regression testing and tuning by validating outputs against a golden dataset.

Core Features & Use Cases

  • Local structure checks with --check to validate setup without API calls.
  • Full regression runs against OpenAI or Gemini using a configured prompt.txt and cases.yaml.
  • Versioned changes tracked in prompt.txt and CHANGELOG.md for auditable updates.
  • Use cases include prompt optimization, regression testing, and TDD-style prompt development.

Quick Start

Run python3 tuner.py --check to verify the local structure, then run python3 tuner.py to execute full regression tests against your configured provider.

Frequently Asked Questions about prompt-tuner

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run prompt regression testing against a golden dataset?▼

Prompt regression testing validates outputs against a golden dataset by running full regression tests against OpenAI or Gemini, requiring a configured prompt.txt and cases.yaml to ensure repeatable improvements.

What is prompt tuning and how does versioning work?▼

Prompt tuning optimizes instructions through regression tests, using versioned changes tracked in prompt.txt and CHANGELOG.md to provide auditable updates for repeatable LLM output improvements.

Can I validate my prompt regression setup without making API calls?▼

You can validate your prompt regression setup without API calls by running the local structure check command, which verifies the configuration of prompt.txt and cases.yaml before executing provider-driven regressions.

Does prompt regression testing work with both OpenAI and Gemini providers?▼

Prompt regression testing works with both OpenAI and Gemini providers, requiring environment-configured API keys to execute full regression runs against the selected model provider.

How do I get started with TDD-style prompt development for LLMs?▼

TDD-style prompt development starts by defining test cases in cases.yaml and an initial prompt.txt, then running local structure checks before executing full regressions to validate output consistency.

Why do I need a golden dataset for LLM prompt optimization?▼

A golden dataset is needed for LLM prompt optimization to provide a stable validation baseline, allowing regression tests to detect output degradations and ensure prompt edits result in auditable improvements.

Related Skills

Explore More Software Engineering Skills→
yu13130122297yu13130122297

scout

Identify uncertainties in research framing, datasets, metrics, and baselines.

Community
Intermediate
giosuetedeschi-specgiosuetedeschi-spec

prompt-engineering

Design and refine prompts for consistent LLM outputs.

Community
Advanced
project-axproject-ax
5

ax-config

Manages AX configuration settings including YAML parsing, path resolution, environment variables, and OAuth token refresh.

Official
Intermediate
cuoreinpacecuoreinpace

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Community
Advanced