skills.rest
bilacchi
bilacchiCOMMUNITY
Explore all skills by this author →
GitHub

llm-evaluation

Evaluate LLM application outputs using automated metrics, LLM-as-judge, and A/B testing.

Updated Jun 12, 2026
Search Tags:#llm-as-judge#benchmarking#llm evaluation#regression testing#a/b testing#rag metrics#bleu
One-click install
npx skills add https://github.com/bilacchi/agents-skills --skill llm-evaluation-bilacchi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/bilacchi/agents-skills/tree/main/skills/llm-evaluation
Command: npx skills add https://github.com/bilacchi/agents-skills --skill llm-evaluation-bilacchi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, scikit-learn, scipy, numpy, openai.

What problem does it solve? Measuring whether an LLM application actually works well is hard: single metrics mislead, regressions slip into production, and prompt changes lack statistical validation. This Skill provides a systematic evaluation framework covering automated metrics, human evaluation, LLM-as-judge, and A/B testing. ## Core Features & Use Cases - Automated Metrics: Compute BLEU, ROUGE, BERTScore, perplexity, classification metrics, and RAG retrieval metrics like MRR and NDCG. - LLM-as-Judge & Human Evaluation: Score outputs with pointwise or pairwise LLM judges, and structure human annotation with inter-rater agreement via Cohen's kappa. - A/B Testing & Regression Detection: Run t-tests with Cohen's d effect sizes and detect metric regressions against baselines before deployment. - Use Case: Before shipping a prompt change, run your test suite through the evaluation harness, compare variants with statistical tests, and block deployment if any metric regresses beyond your threshold. ## Quick Start Ask the AI to evaluate your model's responses against a set of test cases using BLEU, BERTScore, and an LLM judge, then report which prompt variant performs better.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM outputs with automated metrics?▼

Compute BLEU and ROUGE for n-gram overlap, BERTScore for embedding-based semantic similarity, and perplexity for language model confidence. For classification tasks use accuracy, precision, recall, and F1; for RAG use MRR, NDCG, and Precision@K.

What is LLM-as-judge and when should I use it?▼

LLM-as-judge uses a stronger model to score or compare outputs from another model, either pointwise, pairwise, or against a reference. Use it when automated metrics cannot capture qualities like helpfulness or coherence, but validate it against human judgments.

How do I detect LLM performance regressions before deployment?▼

Store baseline metric scores and compare new results against them, flagging any metric whose relative drop exceeds a set threshold such as 5 percent. Integrate this check into your CI/CD pipeline so regressions block releases automatically.

How do I measure inter-rater agreement in human evaluation?▼

Use Cohen's kappa from sklearn to quantify agreement between two annotators on the same rating task. Values below 0.2 indicate slight agreement, 0.4 to 0.6 moderate, and above 0.8 almost perfect agreement.

What are the limitations of relying on a single evaluation metric?▼

Optimizing one metric often degrades others, and small test sets produce unreliable conclusions. Combine multiple metrics, use representative data, apply statistical tests for comparisons, and validate automated scores with human judgment.

Related Skills

Explore More Software Engineering Skills→
davicqueirozdavicqueiroz

RAG Evaluation Skill

Evaluate RAG retrieval, generation, and latency metrics against defined targets.

Community
Advanced
jaimedhenriquesjaimedhenriques

metrics-tracking

Define and analyze product metrics for OKRs and dashboards.

Community
Intermediate
langwatchlangwatch
3.5k

debug-with-langwatch

Diagnose production LLM agent failures by inspecting LangWatch traces, spans, and monitor scores.

Official
Intermediate
markus41markus41

harness-ci

Automates CI/CD pipelines for Kubernetes and other deployments with built-in security scanning and GitOps integration.

Community
Advanced