What problem does it solve? Evaluating and comparing large language models requires standardized prompts, metrics, and benchmarks. This Skill provides reproducible LLM benchmarking across 60+ academic tasks (MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag) so you can measure model quality, compare models, and track training progress with industry-standard methodology. ## Core Features & Use Cases - Standardized Benchmarking: Run MMLU, GSM8K, HumanEval, HellaSwag, ARC, TruthfulQA, and 60+ other tasks with consistent prompts and metrics used by EleutherAI and HuggingFace. - Multi-Backend Support: Evaluate HuggingFace models, vLLM-accelerated inference (5-10x faster), quantized models, and API models (OpenAI, Anthropic, local OpenAI-compatible servers). - Training Progress Tracking: Automate periodic checkpoint evaluation during training and plot learning curves from JSON results. - Distributed Evaluation: Scale across GPUs with data parallelism, tensor parallelism, and pipeline parallelism for models up to 70B+ parameters. - Use Case: Compare Llama-2-7B, Mistral-7B, and Phi-2 on MMLU, GSM8K, HellaSwag, and TruthfulQA, then generate a markdown comparison table of results. ## Quick Start Ask the AI to evaluate a HuggingFace model like meta-llama/Llama-2-7b-hf on MMLU and GSM8K with 5-shot prompting and save the results to a JSON file.
