What problem does it solve? Benchmarking language models consistently is difficult because results vary with prompts, few-shot settings, and evaluation code. This Skill standardizes LLM evaluation across 60+ academic benchmarks (MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag) so results are reproducible and comparable across models, checkpoints, and papers. ## Core Features & Use Cases - Standardized Benchmark Suites: Run MMLU, GSM8K, HellaSwag, ARC, TruthfulQA, HumanEval, and more with one command against HuggingFace, vLLM, or API-based models. - Training Progress Tracking: Evaluate checkpoints periodically during training and plot learning curves from saved JSON results. - Model Comparison Tables: Batch-evaluate multiple models and generate side-by-side comparison tables for papers or release decisions. - Distributed & API Evaluation: Scale with vLLM tensor/data parallelism, multi-GPU accelerate launches, or evaluate OpenAI and Anthropic API models. - Use Case: Before releasing a fine-tuned 7B model, run the standard suite (mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge) with 5-shot prompting, then compare scores against Llama-2 and Mistral baselines in a markdown table. ## Quick Start Ask the AI to evaluate your HuggingFace model on MMLU, GSM8K, and HellaSwag with 5-shot prompting and save the results to a JSON file.
