lm-evaluation-harness

What problem does it solve?

Provides a unified interface to evaluate language models across a broad suite of benchmarks (MMLU, GSM8K, HumanEval, etc.), enabling rigorous model comparison.

Core Features & Use Cases

• Benchmark Suite: 60+ tasks including code, reasoning, and multilingual benchmarks
• Model Compatibility: supports HuggingFace, vLLM, API-based models
• Distributed / Parallel Runs: scalable evaluation on GPUs or CPUs
• Custom Tasks: add private datasets and metrics

Quick Start

Evaluate a 7B HF model on GSM8K and HumanEval using 5-shot prompts.