skills.rest
ACGSpgp
ACGSpgpOFFICIAL
Explore all skills by this author →
WebsiteGitHub

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Updated Aug 23, 2026
Search Tags:#testing#quality assurance#llm-as-judge#llm#benchmarking#metrics#evaluation
One-click install
npx skills add https://github.com/ACGSpgp/ACGS --skill llm-evaluation-acgspgp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-evaluation
Source: https://github.com/ACGSpgp/ACGS/tree/main/.agent/workflows/llm-application-dev/1.2.1/skills/llm-evaluation
Command: npx skills add https://github.com/ACGSpgp/ACGS --skill llm-evaluation-acgspgp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nltk, rouge-score, bert-score, transformers, detoxify, openai, scipy, numpy, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a comprehensive framework for evaluating the performance and quality of Large Language Model (LLM) applications, ensuring reliability and identifying areas for improvement.

Core Features & Use Cases

  • Automated Metrics: Implement standard metrics like BLEU, ROUGE, BERTScore, Accuracy, Precision, Recall, F1, MRR, NDCG, and more for quantitative assessment.
  • Human Evaluation: Define dimensions and guidelines for manual assessment of aspects like accuracy, coherence, relevance, fluency, safety, and helpfulness.
  • LLM-as-Judge: Leverage powerful LLMs to evaluate outputs through pointwise or pairwise comparisons, with or without reference data.
  • A/B Testing & Regression: Conduct statistical analysis for A/B tests and implement regression detection to monitor performance over time.
  • Benchmarking: Run models against benchmark datasets to establish performance baselines.
  • Use Case: You have developed a new LLM-powered chatbot. Use this Skill to systematically measure its response quality using a combination of automated metrics and human feedback, compare it against a previous version, and ensure no performance regressions have been introduced.

Quick Start

Use the llm-evaluation skill to evaluate your model by defining an evaluation suite with accuracy and BLEU metrics and running it against a list of test cases.

Frequently Asked Questions about llm-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM application performance using automated metrics?▼

To evaluate LLM application performance, you can implement automated metrics like BLEU, ROUGE, BERTScore, Accuracy, Precision, Recall, F1, MRR, and NDCG. You define an evaluation suite with your chosen metrics and run it against a list of test cases to quantitatively assess response quality.

Can I use LLM-as-Judge to evaluate model outputs without reference data?▼

LLM-as-Judge evaluation works by leveraging powerful LLMs to evaluate model outputs through pointwise or pairwise comparisons. This approach can be conducted with or without reference data to systematically measure AI application quality and response accuracy.

What is the best way to detect performance regressions when testing LLM applications?▼

The best way to detect performance regressions is by conducting statistical analysis for A/B tests and implementing regression detection. This monitors your LLM application's performance over time, ensuring new versions introduce no degradations compared to previous baselines.

How do I measure LLM safety and helpfulness using human evaluation?▼

To measure LLM safety and helpfulness, you can define dimensions and guidelines for manual human evaluation. This facilitates manual assessment of aspects like accuracy, coherence, relevance, fluency, safety, and helpfulness alongside automated metrics.

How do I establish performance baselines for my LLM model?▼

To establish performance baselines, you run your LLM models against benchmark datasets. This benchmarking process provides a foundational performance measurement, allowing you to systematically measure AI application quality and compare future iterations.

Do I need specific Python libraries to run BERTScore and ROUGE for LLM evaluation?▼

To use BERTScore and ROUGE for LLM evaluation, you need dependencies like nltk, rouge-score, bert-score, transformers, scipy, and numpy. These libraries facilitate the quantitative assessment of text generation quality within your testing framework.

Related Skills

Explore More Software Engineering Skills→
surus-latsurus-lat
8

setup-data

Configure and validate benchmark datasets by mapping CSV or JSONL files to required schemas.

Official
Intermediate
vuralserhat86vuralserhat86
43

llm_evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Community
Advanced
JNZaderJNZader

llm-evaluation

Automate LLM output evaluation with BLEU, ROUGE, BERTScore, and judge prompts.

Community
Advanced
darkounus90darkounus90
1

pair-trade-screener

Identify and analyze stock pairs using cointegration and correlation tests.

Community
Advanced