skills.rest
rigasnameji
rigasnamejiCOMMUNITY
Explore all skills by this author →
GitHub

prompt-engineer-toolkit

Tests, scores, and versions LLM prompts with A/B evaluation and JSONL history tracking.

Updated May 18, 2026
Search Tags:#prompt engineering#llm evaluation#regression testing#a/b testing#prompt templates#prompt versioning#marketing ai
One-click install
npx skills add https://github.com/rigasnameji/address_uluwatu --skill prompt-engineer-toolkit-rigasnameji
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: prompt-engineer-toolkit
Source: https://github.com/rigasnameji/address_uluwatu/tree/main/ADDRESS/marketing-skill/prompt-engineer-toolkit
Command: npx skills add https://github.com/rigasnameji/address_uluwatu --skill prompt-engineer-toolkit-rigasnameji

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Prompts often live as ad-hoc drafts with no way to measure quality, compare variants, or track changes over time. This Skill turns prompts into versioned, testable assets so you can pick winners with evidence instead of intuition and catch regressions before deployment. ## Core Features & Use Cases - A/B Prompt Evaluation: Run two prompt variants against structured JSON test cases with per-case scoring for expected content, forbidden content, regex compliance, and output length. - Prompt Versioning: Store immutable prompt history in a local JSONL file with add, list, diff, and changelog commands, including author and change notes. - Templates and Rubrics: Use ready-made prompt templates (classifier, extractor, summarizer, and more) plus an evaluation rubric with acceptance gates. - Use Case: Before rolling out a new support ticket classifier prompt, run it against your baseline with realistic test cases, confirm the average score improves with zero forbidden-content violations, then version and promote it. ## Quick Start Ask the AI to A/B test two prompt files against a JSON test case suite using scripts/prompt_tester.py and report the winning variant with scores.

Frequently Asked Questions about prompt-engineer-toolkit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I A/B test two prompts against test cases?▼

Run scripts/prompt_tester.py with --prompt-a-file, --prompt-b-file, and --cases-file pointing to a JSON array of test cases. Each case defines input, expected_contains, forbidden_contains, and expected_regex, and the script scores both variants and reports a winner.

How do I track prompt version history and diffs?▼

Use scripts/prompt_versioner.py with the add, list, diff, and changelog commands. Versions are stored in a local JSONL file with author, timestamp, and change note, and diff produces a unified diff between any two versions.

Can I test prompts against a live LLM instead of static scoring?▼

Yes, pass --runner-cmd with a command template containing {prompt} and {input} placeholders, such as a CLI call to your model. Without a runner command, the script performs static prompt quality scoring only.

What test case format does the prompt tester require?▼

Test cases are a JSON array where each case has an input string plus optional expected_contains, forbidden_contains, and expected_regex arrays. Scoring penalizes missed expected content and forbidden hits while rewarding regex matches.

What are the limitations of static prompt scoring?▼

Static mode only renders the prompt with the case input substituted, so it cannot measure actual model behavior. For meaningful quality signals, use a runner command to execute real model outputs before comparing variants.

Related Skills

Explore More Marketing & Sales Skills→
ssdeanxssdeanx
43

test-cases

Generate structured test cases with traceability from PRD requirements.

Community
Advanced
mgher668mgher668

canary

Monitor production URLs for console errors, performance regressions, and page failures.

Community
Advanced
zhuangbiaoweizhuangbiaowei
2

nemo-guardrails

Implement runtime safety guardrails for LLM applications using NeMo Guardrails and Colang 2.0 DSL.

Community
Advanced
zainhaszainhas
2

together-evaluations

Benchmark LLM outputs with Together AI's Evals across serverless, dedicated, and external targets.

Community
Advanced