skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Updated Mar 23, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/hanasho744/codex --skill evaluating-llms-harness-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/lm-evaluation-harness
Command: npx skills add https://github.com/hanasho744/codex --skill evaluating-llms-harness-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Benchmarking language models consistently is difficult because results vary with prompts, few-shot settings, and evaluation code. This Skill standardizes LLM evaluation across 60+ academic benchmarks so results are reproducible and comparable across models, checkpoints, and publications. ## Core Features & Use Cases - Standardized Benchmark Suites: Run MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, and 60+ other tasks with one command using community-standard prompts and metrics. - Multiple Model Backends: Evaluate HuggingFace models, vLLM-served models (5-10x faster), quantized checkpoints, and API models from OpenAI and Anthropic. - Training Progress Tracking: Evaluate checkpoints periodically during training and plot learning curves to monitor model quality over time. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU, GSM8K, HellaSwag, and TruthfulQA, then generate a markdown comparison table for a model release report. ## Quick Start Ask the AI to evaluate a HuggingFace model like meta-llama/Llama-2-7b-hf on MMLU, GSM8K, and HellaSwag with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a HuggingFace model on MMLU?▼

Run lm_eval with --model hf, --model_args pretrained=your-model, --tasks mmlu, and --num_fewshot 5 for the standard 5-shot setting. Results with accuracy and standard error are saved as JSON when you specify --output_path.

How can I make LLM benchmark evaluation faster?▼

Use the vLLM backend with --model vllm instead of hf, which runs 5-10x faster through optimized inference. You can also reduce few-shot examples, use --batch_size auto, or evaluate MMLU subsets like mmlu_stem instead of all 57 subjects.

Can lm-evaluation-harness evaluate OpenAI or Anthropic API models?▼

Yes, it supports openai-chat-completions and anthropic-chat model types after setting the OPENAI_API_KEY or ANTHROPIC_API_KEY environment variable. Chat APIs lack logprobs, so they only work with generation tasks like MMLU and GSM8K, not perplexity tasks.

Why do my benchmark results differ from published numbers?▼

Mismatches usually come from wrong few-shot counts (most papers use 5-shot), incorrect task names, or a tokenizer that does not match the model. Verify --num_fewshot, the exact task identifier, and that pretrained and tokenizer arguments point to the same model.

What should I do when evaluation runs out of GPU memory?▼

Reduce batch size to 1 or use --batch_size auto, enable 8-bit quantization with load_in_8bit=True, or shard large models across GPUs with parallelize=True. For 70B models, use vLLM with tensor_parallel_size set to your GPU count.

When should I use HELM or MT-Bench instead of lm-evaluation-harness?▼

Use HELM when you need broader evaluation covering fairness, efficiency, and calibration, and MT-Bench for multi-turn conversational quality judged by LLMs. lm-evaluation-harness is best for standardized academic benchmarks with reproducible prompts and metrics.

Related Skills

Explore More Data & Analytics Skills→
i-bebsii-bebsi

evaluating-llms-harness

Benchmarks LLMs across 60+ academic tasks like MMLU, GSM8K, and HumanEval.

Community
Advanced
DTMC-marketplaceDTMC-marketplace

model-selection

Guide AI/ML model selection through capability matching, benchmarks, cost analysis, and deployment constraints.

Official
Advanced
NVIDIA-NeMoNVIDIA-NeMo
58

nemo-evaluator

Standardize LLM agent and RAG pipeline evaluation with composable metrics and reproducible benchmarks.

Official
Advanced
haJ1thaJ1t

model-evaluator

Evaluate AI models on quality, latency, cost, safety, and fairness benchmarks.

Community
Advanced