skills.rest
i-bebsi
i-bebsiCOMMUNITY
Explore all skills by this author →
GitHub

evaluating-llms-harness

Benchmarks LLMs across 60+ academic tasks like MMLU, GSM8K, and HumanEval.

Updated Jun 17, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/i-bebsi/hermes-agent --skill evaluating-llms-harness-i-bebsi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/i-bebsi/hermes-agent/tree/main/hermes-config/skills/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/i-bebsi/hermes-agent --skill evaluating-llms-harness-i-bebsi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating and comparing language models requires standardized prompts, metrics, and benchmarks; running ad-hoc tests produces inconsistent, non-reproducible results that cannot be compared across models or papers. ## Core Features & Use Cases - Standardized Benchmarking: Evaluate any HuggingFace, vLLM, or API-based model on 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, and HellaSwag with industry-standard few-shot settings. - Training Progress Tracking: Run periodic evaluations on checkpoints during training and plot learning curves to monitor model quality over time. - Model Comparison: Batch-evaluate multiple models and generate comparison tables for release decisions or academic reporting. - Use Case: A team fine-tuning a 7B model runs MMLU and GSM8K on each checkpoint via vLLM, then compares scores against Llama 2 and Mistral baselines to decide which checkpoint to release. ## Quick Start Ask the agent to evaluate a HuggingFace model on MMLU and GSM8K using lm_eval with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark an LLM on MMLU with lm-evaluation-harness?▼

Run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5 for the standard 5-shot setting. Results with accuracy and standard error are saved to a JSON output file.

How can I speed up LLM evaluation on benchmarks?▼

Use the vLLM backend with --model vllm and tensor_parallel_size set to your GPU count, which is 5-10x faster than the HuggingFace backend. You can also reduce few-shot examples or evaluate MMLU subsets like mmlu_stem.

Can lm-evaluation-harness evaluate OpenAI or Anthropic API models?▼

Yes, it supports openai-chat-completions and anthropic-chat model types using your API keys. Chat APIs lack logprobs, so they only work with generation tasks like MMLU and GSM8K, not perplexity tasks.

Why does evaluation run out of memory on large models?▼

A 70B model in bf16 needs roughly 168GB, exceeding a single GPU. Use parallelize=True for HuggingFace model sharding, vLLM tensor parallelism, 8-bit quantization, or reduce batch size.

When should I use HELM or MT-Bench instead of lm-evaluation-harness?▼

Use HELM for broader evaluation covering fairness, efficiency, and calibration, or MT-Bench for multi-turn conversational assessment. lm-evaluation-harness is best for standardized academic benchmarks and reproducible model comparisons.

Related Skills

Explore More Data & Analytics Skills→
heysuhasheysuhas
2

axolotl

Manage YAML-based configuration for fine-tuning LLMs with PyTorch and HuggingFace.

Community
Advanced
surus-latsurus-lat
8

push-to-latamboard

Merge local benchmark results into the HuggingFace LatamBoard leaderboard dataset.

Official
Advanced
ACGSpgpACGSpgp

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Official
Advanced
haJ1thaJ1t

model-evaluator

Evaluate AI models on quality, latency, cost, safety, and fairness benchmarks.

Community
Advanced