evaluating-llms-harness

Run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5 for the standard 5-shot setting. Results with accuracy and standard error are saved to a JSON output file.

Use the vLLM backend with --model vllm and tensor_parallel_size set to your GPU count, which is 5-10x faster than the HuggingFace backend. You can also reduce few-shot examples or evaluate MMLU subsets like mmlu_stem.

Yes, it supports openai-chat-completions and anthropic-chat model types via API keys. Chat APIs lack logprobs, so they only work with generation tasks like MMLU and GSM8K, not perplexity tasks.

Check that your few-shot count matches the paper (usually 5-shot), verify the exact task name, and confirm the model and tokenizer match. Sampling temperature and seed settings also affect generation-based tasks.

Reduce batch size or use --batch_size auto, enable 8-bit or 4-bit quantization via load_in_8bit in model_args, or shard large models across GPUs with parallelize=True or vLLM tensor parallelism.

Use HELM for broader evaluation covering fairness, efficiency, and calibration, or MT-Bench for multi-turn conversational assessment. lm-evaluation-harness is best for standardized academic benchmarks with reproducible prompts.