skills.rest
BryanPinheiro77
BryanPinheiro77COMMUNITY
Explore all skills by this author →
GitHub

senior-prompt-engineer

Analyze, optimize, and evaluate prompts, RAG pipelines, and agent configurations with stdlib Python tools.

2|Updated Mar 24, 2026
Search Tags:#prompt engineering#llm evaluation#few-shot#eval-driven#token estimation#agent validation#rag metrics
One-click install
npx skills add https://github.com/BryanPinheiro77/FinanceBot-BackEnd --skill senior-prompt-engineer-bryanpinheiro77
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: senior-prompt-engineer
Source: https://github.com/BryanPinheiro77/FinanceBot-BackEnd/tree/main/.agents/skills/senior-prompt-engineer
Command: npx skills add https://github.com/BryanPinheiro77/FinanceBot-BackEnd --skill senior-prompt-engineer-bryanpinheiro77

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Prompt iteration without measurement is guesswork: teams tweak prompts based on vibes, ship regressions, and misdiagnose retrieval failures as prompt problems. This Skill enforces eval-driven prompt engineering by capturing baselines, comparing revisions against them, and gating changes on measurable metrics before anything ships. ## Core Features & Use Cases - Prompt Static Analysis: Score prompts for clarity and structure (0-100), detect ambiguous verbs and redundant phrasing, estimate tokens, and extract few-shot examples via scripts/prompt_optimizer.py. - RAG Quality Measurement: Compute context relevance, precision@k, coverage, faithfulness, and groundedness from retrieval and question JSON files with scripts/rag_evaluator.py, so you fix retrieval before rewriting generation prompts. - Agent Config Validation: Validate YAML/JSON agent definitions for tool wiring, missing config, loop risk, and token budgets, with ASCII or Mermaid workflow visualization via scripts/agent_orchestrator.py. - Use Case: Before shipping a revised system prompt, run the optimizer to save a baseline JSON, apply one change at a time, then compare the revision against the baseline and require clarity to hold while tokens stay within 10% — blocking regressions automatically. ## Quick Start Analyze my prompt file current_prompt.txt, save a baseline report, and tell me which issues to fix first before I revise it.

Frequently Asked Questions about senior-prompt-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize a prompt without regressing quality?▼

Capture a baseline first with the prompt optimizer's --analyze --json --output flags, then apply one change at a time and compare each revision against the baseline. Gate shipping on clarity not dropping and token count staying within 10% of baseline, plus no previously-passing eval case failing.

How do I measure RAG retrieval quality?▼

Run the RAG evaluator with a contexts JSON file and a questions JSON file to get context relevance, precision@k, coverage, faithfulness, and groundedness scores. Treat relevance below 0.80 as a retrieval problem — fix chunking, embeddings, or filters before rewriting the generation prompt.

How many few-shot examples should a prompt have?▼

Start with zero examples and measure, since current models often need none. Add examples only for failure clusters the eval set reveals, capped at 3-5, ordered simple to edge case to a negative example showing what not to do.

Does this work with any LLM provider or model?▼

Yes, the tools are model-agnostic by design and never hardcode model IDs or pricing. The --model flag only infers tokenizer family for token estimation, and you supply your provider's current per-million-token rates via price flags when you want dollar figures.

Why does my agent config fail validation?▼

Common causes are duplicate tool names, tools missing descriptions or required config entries, and max_iterations above 50 which flags infinite-loop risk. Run the orchestrator with --validate to get a report listing every error and warning to fix.

When should I use structured outputs instead of prompt instructions?▼

Prefer API-native structured outputs or JSON schema enforcement whenever the provider supports them, since they guarantee output shape while prompt text cannot. Use schema-in-prompt with an explicit example only as a fallback for APIs without schema support.

Related Skills

Explore More Software Engineering Skills→
tanveerriaztanveerriaz
1

llm-eval-golden-set

Builds a golden set of test cases and an eval runner to detect LLM output regressions.

Community
Intermediate
PiercingXXPiercingXX
1

ops-canary

Monitor post-deploy canary signals and trigger rollback on quality degradation.

Community
Intermediate
ContextJet-aiContextJet-ai
29

collect-user-feedback

Collect user feedback on LLM outputs and integrate it with observability tools.

Official
Intermediate
lebsrallebsral
11

ai-fine-tuning

Fine-tune models on labeled datasets using BootstrapFinetune workflows.

Community
Advanced