skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

evaluating-llms-harness

Benchmarks LLMs across 60+ academic tasks using standardized prompts and metrics.

1|Updated Mar 12, 2026
Search Tags:#benchmarking#llm evaluation#model comparison#vllm#gsm8k#mmlu#lm-eval
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill evaluating-llms-harness-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-llms-harness
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/evaluation/lm-evaluation-harness
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill evaluating-llms-harness-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires lm-eval, transformers, vllm, and includes references (resource) components.

What problem does it solve? Evaluating language models consistently is hard: different prompts, few-shot settings, and metrics produce incomparable results. This Skill runs standardized academic benchmarks (MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, and 60+ more) so you can compare models, track training progress, and report reproducible numbers. ## Core Features & Use Cases - Standardized Benchmark Suites: Run MMLU, GSM8K, HellaSwag, ARC, TruthfulQA, HumanEval, and more with one command using the same prompts and metrics used by EleutherAI and HuggingFace leaderboards. - Multiple Model Backends: Evaluate HuggingFace checkpoints, quantized models, vLLM-served models (5-10x faster), and API models like GPT-4 and Claude through a unified interface. - Training Progress Tracking: Automate periodic checkpoint evaluation during training and plot learning curves from saved JSON results. - Use Case: You are preparing a model release and need a comparison table. Run the standard suite across Llama-2-7B, Mistral-7B, and Phi-2, then generate a Markdown table of MMLU, GSM8K, HellaSwag, and TruthfulQA scores from the JSON outputs. ## Quick Start Ask the agent to evaluate a HuggingFace model on MMLU, GSM8K, and HellaSwag with 5-shot prompting and save the results to a JSON file.

Frequently Asked Questions about evaluating-llms-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an LLM on MMLU and GSM8K?▼

Run lm_eval with the hf backend, passing the model path and a comma-separated task list such as mmlu,gsm8k,hellaswag. Use --num_fewshot 5 for the standard 5-shot setting and --output_path to save results as JSON.

How can I make lm-evaluation-harness run faster?▼

Use the vLLM backend instead of HuggingFace for 5-10x faster inference, reduce few-shot examples to zero, or evaluate task subsets like mmlu_stem. Multi-GPU data parallelism with accelerate also gives near-linear speedups.

Can I evaluate OpenAI or Anthropic API models with lm-eval?▼

Yes, use the openai-chat-completions or anthropic-chat model types with your API key exported. Note that chat APIs lack logprobs, so they only support generation tasks like MMLU and GSM8K, not perplexity tasks.

Why do my benchmark results differ from published numbers?▼

Mismatches usually come from different few-shot counts, wrong task variants, or tokenizer mismatches. Verify --num_fewshot matches the paper (usually 5), use the exact task name like mmlu, and confirm the tokenizer matches the model.

How do I evaluate a model that does not fit on one GPU?▼

For HuggingFace models, pass parallelize=True to shard weights across GPUs. For vLLM, set tensor_parallel_size to split the model, and combine with data_parallel_size for additional throughput on large models like 70B.

When should I use HELM or MT-Bench instead of lm-evaluation-harness?▼

Use HELM for broader evaluations covering fairness, efficiency, and calibration, or MT-Bench and AlpacaEval for multi-turn conversational and instruction-following quality judged by LLMs. lm-evaluation-harness is best for standardized academic accuracy benchmarks.

Related Skills

Explore More Data & Analytics Skills→
ScientiaCapitalScientiaCapital
1

model-deployment

Export fine-tuned models to GGUF and deploy with vLLM or Ollama.

Community
Advanced
surus-latsurus-lat
8

push-to-latamboard

Merge local benchmark results into the HuggingFace LatamBoard leaderboard dataset.

Official
Advanced
simplysmartaisimplysmartai

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and LLM-as-judge patterns.

Community
Advanced
haJ1thaJ1t

model-evaluator

Evaluate AI models on quality, latency, cost, safety, and fairness benchmarks.

Community
Advanced