eval-bench

What problem does it solve?

This Skill automates the rigorous evaluation of AI models, ensuring their performance, safety, and reliability before deployment.

Core Features & Use Cases

• Model Evaluation: Run comprehensive benchmarks using lm-evaluation-harness covering common NLP tasks (MMLU, HellaSwag, ARC).
• Safety & Bias Testing: Assess models for truthfulness, toxicity, bias, and stereotyping with benchmarks like TruthfulQA, ToxiGen, BBQ, and CrowS-Pairs.
• Code Generation Evaluation: For code models, evaluate performance on HumanEval, MBPP, and MultiPL-E.
• RAG & Perplexity: Evaluate Retrieval-Augmented Generation quality with Ragas and measure perplexity on held-out datasets.
• Reporting: Generate human-readable reports summarizing findings and providing release readiness recommendations.

Quick Start

Run a full model evaluation using the eval-bench skill with default settings.