skills.rest
agenticgogol
agenticgogolCOMMUNITY
Explore all skills by this author →
GitHub

baseline-runner

Generates promptfoo configs and runs evals to produce scored baseline results.

1|9|Updated Jul 11, 2026
Search Tags:#llm evaluation#baseline#promptfoo#eval#golden dataset#ragas#llm-rubric
One-click install
npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill baseline-runner-agenticgogol
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: baseline-runner
Source: https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul/tree/main/.claude/skills/baseline-runner
Command: npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill baseline-runner-agenticgogol

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running an LLM agent evaluation requires keeping the promptfoo config in sync with a changing golden set, graders, and judge prompts, and hand-written configs drift out of date quickly. This Skill automates config generation and eval execution so you get reproducible, scored baseline results. ## Core Features & Use Cases - Script-Generated Config: Writes and runs a generator script that reads golden_set.jsonl, graders.md, and judge_prompts/ to emit eval/promptfooconfig.yaml, keeping the config in sync automatically. - Ordered Assertions: Runs cheap code-based assertions before expensive llm-rubric judge calls so failing examples short-circuit early. - RAG Metric Support: Detects RAG-shaped agents and generates a Ragas config covering context precision, recall, and faithfulness. - Use Case: After building judge prompts for a support-chat agent, run the eval to produce eval/results/baseline.json with aggregate, per-task, and per-metric scores, then hand failing examples to failure analysis. ## Quick Start Run the eval against my golden set and generate the promptfoo config, then write the scored baseline results.

Frequently Asked Questions about baseline-runner

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a promptfoo eval against a golden dataset?▼

Generate the promptfoo config from your golden_set.jsonl, graders.md, and judge_prompts/ using a generator script, then execute the eval to write results to eval/results/baseline.json. Generating the config via script keeps it in sync as the golden set grows.

How do I keep a promptfoo config in sync with changing test cases?▼

Write a generator script that reads the golden set, graders, and judge prompts and emits promptfooconfig.yaml, rather than hand-editing the YAML. Regenerating from the script prevents drift when graders or examples change.

Should code-based or LLM-rubric assertions run first in an eval?▼

Code-based assertions should run first because they are cheap and deterministic, letting failing examples short-circuit before spending judge calls. Only examples that need model-based judgment should consume llm-rubric calls.

Does promptfoo support RAG metrics like faithfulness and context recall?▼

For RAG-shaped agents, generate a separate Ragas config covering context precision, context recall, and faithfulness alongside the promptfoo eval. Detect the RAG shape from the eval tasks and metrics definitions first.

Why should I confirm cost before running an LLM eval?▼

Evals multiply examples by judge calls, so a live run against a paid provider can incur significant spend. State the number of examples times judge calls and get explicit approval before executing a real run.

Related Skills

Explore More Software Engineering Skills→
tanveerriaztanveerriaz
1

llm-eval-golden-set

Builds a golden set of test cases and an eval runner to detect LLM output regressions.

Community
Intermediate
HomericIntelligenceHomericIntelligence
18

analyze-ci-failure-logs

Parse and categorize CI failure logs to identify root causes.

Official
Intermediate
mberto10mberto10

langfuse-score-analytics

Analyze Langfuse scores for trends, comparisons, and regression detection.

Community
Intermediate
allankltsnallankltsn

agno-testing-strategy

Test and optimize Agno agents with deterministic fakes, evals, and parameter search.

Community
Advanced