skills.rest
swcstudiospace
swcstudiospaceOFFICIAL
Explore all skills by this author →
Website𝕏@govantarcticaGitHub

evaluating-llms-harness

Benchmarks LLMs across 60+ academic tasks using the lm-evaluation-harness framework.

Updated Aug 19, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/swcstudiospace/aimeecodes --skill evaluating-llms-harness-swcstudiospace
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/swcstudiospace/aimeecodes/tree/main/.aimee/skills/evaluating-llms-harness
Command: npx skills add https://github.com/swcstudiospace/aimeecodes --skill evaluating-llms-harness-swcstudiospace

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating and comparing language models requires standardized prompts, metrics, and benchmarks; running ad-hoc tests produces inconsistent, non-reproducible results that cannot be compared across models or papers. ## Core Features & Use Cases - Standardized Benchmarking: Run 60+ academic benchmarks (MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC) with consistent prompts and metrics used by EleutherAI and HuggingFace. - Flexible Model Backends: Evaluate HuggingFace models, vLLM-served models (5-10x faster), quantized checkpoints, and API models like GPT-4 and Claude. - Training Progress Tracking: Automate periodic checkpoint evaluation during training and plot learning curves over time. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU, GSM8K, and HellaSwag, then generate a markdown comparison table of accuracy scores for a model selection decision. ## Quick Start Ask the agent to evaluate a HuggingFace model on MMLU and GSM8K with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark an LLM on MMLU with lm-evaluation-harness?▼

Install lm-eval via pip, then run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5. Results with accuracy and standard error are saved to a JSON output file.

What is the difference between lm-evaluation-harness and HELM?▼

lm-evaluation-harness focuses on standardized academic benchmarks like MMLU and GSM8K with reproducible prompts. HELM from Stanford covers broader dimensions including fairness, efficiency, and calibration. Use lm-eval for standard model quality comparisons.

Can lm-eval evaluate OpenAI or Anthropic API models?▼

Yes, it supports openai-chat-completions and anthropic-chat model types via API keys. Note that chat APIs without logprobs can only run generation tasks like MMLU and GSM8K, not perplexity or loglikelihood tasks.

Why is my MMLU evaluation so slow and how do I speed it up?▼

Full MMLU takes about 2 hours on a 7B model with the HuggingFace backend. Switch to the vLLM backend for 5-10x speedup, reduce num_fewshot to 0, or evaluate only a subset like mmlu_stem.

Why does HumanEval fail to run in lm-eval?▼

Code-executing tasks like HumanEval and MBPP require the explicit --confirm_run_unsafe_code flag because they execute generated code. Without this flag, lm-eval refuses to run the task rather than silently skipping execution.

How much GPU memory do I need to evaluate a 70B model?▼

A 70B model in bf16 needs about 140GB, so it will not fit on a single GPU. Use tensor parallelism with vLLM (tensor_parallel_size=4 or 8) or HuggingFace parallelize=True to shard the model across multiple GPUs.

Related Skills

Explore More Data & Analytics Skills→
kaminocorpkaminocorp
1

fine-tuning-with-trl

Fine-tune language models with TRL using SFT, DPO, PPO, and GRPO training methods.

Official
Advanced
Andrew-GirgisAndrew-Girgis

huggingface-best

Recommend top HuggingFace models by benchmark scores and device constraints.

Community
Advanced
simplysmartaisimplysmartai

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and LLM-as-judge patterns.

Community
Advanced
rebeatyrebeaty
1

benchmark-onboarder

Convert creativity benchmarks into HELM Scenarios with extracted prompts and mapped fields.

Community
Advanced