What problem does it solve? Running LLM benchmarks consistently across different harnesses, environments, and infrastructure is fragmented and hard to reproduce. This Skill provides a unified workflow for evaluating models against 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) using NVIDIA's NeMo Evaluator with containerized, reproducible execution. ## Core Features & Use Cases - Multi-benchmark evaluation: Run academic, coding, safety, function-calling, and vision-language benchmarks against any OpenAI-compatible endpoint (vLLM, TRT-LLM, NVIDIA NIM). - Multi-backend execution: Execute evaluations locally with Docker, on Slurm HPC clusters with model deployment, or on Lepton cloud, with job status tracking and result export to MLflow, W&B, or JSON. - Model comparison: Benchmark multiple models on identical tasks using Hydra config overrides, then export and compare results. - Use Case: A research team wants to compare Llama 3.1 8B and Mistral 7B on MMLU-Pro, GSM8K, and IFEval. They create one base config, run two evaluations with model overrides, and export both result sets to MLflow for side-by-side comparison. ## Quick Start Ask the AI to evaluate a model on standard benchmarks like MMLU and GSM8K using NeMo Evaluator with a local Docker execution config pointing at your OpenAI-compatible endpoint.
