skills.rest
TechCorp25
TechCorp25COMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Implement evaluation frameworks for LLM applications using automated metrics, LLM-as-judge, and A/B testing.

Updated May 20, 2026
Search Tags:#llm-as-judge#benchmarking#llm evaluation#regression testing#a/b testing#bertscore#langsmith
One-click install
npx skills add https://github.com/TechCorp25/kingdom --skill llm-evaluation-techcorp25
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/TechCorp25/kingdom/tree/main/.claude/skills/llm-application-dev/skills/llm-evaluation
Command: npx skills add https://github.com/TechCorp25/kingdom --skill llm-evaluation-techcorp25

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, anthropic, pydantic, scikit-learn, scipy, numpy, langsmith.

What problem does it solve? Measuring whether an LLM application actually works well is hard: manual spot-checks don't scale, regressions slip into production unnoticed, and comparing models or prompts lacks rigor. This Skill provides a complete evaluation toolkit covering automated metrics, human evaluation, LLM-as-judge patterns, and statistical A/B testing. ## Core Features & Use Cases - Automated Metrics: Implement BLEU, ROUGE, BERTScore, perplexity, and custom metrics like groundedness, toxicity, and factuality using NLI models. - LLM-as-Judge: Use Claude for pointwise scoring, pairwise comparison, and reference-based evaluation with structured Pydantic outputs. - Human Evaluation & A/B Testing: Build annotation frameworks with inter-rater agreement (Cohen's kappa) and run statistically rigorous A/B tests with t-tests and Cohen's d effect sizes. - Regression Detection & Benchmarking: Compare new results against baselines to catch performance regressions, and integrate with LangSmith for dataset-driven evaluation. - Use Case: Before deploying a prompt change to your RAG chatbot, run the evaluation suite against your test dataset, compare results to the baseline with the regression detector, and confirm the improvement is statistically significant via the A/B testing framework. ## Quick Start Ask the AI to set up an evaluation suite with accuracy, BERTScore, and a groundedness metric, then run it against your test cases and flag any regressions versus your baseline.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM application performance automatically?▼

Build an EvaluationSuite with metrics like accuracy, BLEU, and BERTScore, then run it against a list of test cases containing inputs and expected outputs. The suite averages scores across all cases and returns both aggregate metrics and raw per-example scores.

What is LLM-as-judge and when should I use it?▼

LLM-as-judge uses a stronger model like Claude to score or compare outputs from another model. Use it when automated metrics miss quality dimensions like helpfulness or coherence, choosing pointwise scoring, pairwise comparison, or reference-based evaluation depending on whether you have gold-standard answers.

BLEU vs ROUGE vs BERTScore: which metric should I use?▼

BLEU measures n-gram overlap and suits translation tasks, ROUGE is recall-oriented and fits summarization, while BERTScore uses embedding similarity for semantic closeness. For most LLM applications, BERTScore correlates better with human judgment than surface-level n-gram metrics.

How do I detect LLM performance regressions before deployment?▼

Store baseline metric results and use a RegressionDetector that compares new scores against the baseline with a relative-change threshold, typically 5 percent. Any metric dropping beyond the threshold is flagged as a regression before the change ships.

How do I know if an A/B test result is statistically significant?▼

Run an independent t-test on the score distributions of both variants and check that the p-value falls below your alpha, usually 0.05. Also report Cohen's d to quantify effect size, since statistical significance alone does not indicate a meaningful improvement.

Can I integrate LLM evaluation with LangSmith?▼

Yes, create a LangSmith dataset with input-output examples, attach evaluators like qa, context_qa, and cot_qa, then run the evaluate function against your chain. Results include aggregate metrics and experiment metadata for tracking versions over time.

Related Skills

Explore More Software Engineering Skills→
tanveerriaztanveerriaz
1

llm-eval-golden-set

Builds a golden set of test cases and an eval runner to detect LLM output regressions.

Community
Intermediate
Owl-ListenerOwl-Listener
157

longitudinal-measurement

Track AI product quality metrics across releases to detect drift.

Community
Intermediate
nexus-labs-automationnexus-labs-automation
7

llm-call-tracing

Trace LLM API calls to capture model, latency, tokens, cost, and success status.

Official
Advanced
simplysmartaisimplysmartai

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and LLM-as-judge patterns.

Community
Advanced