skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks using containerized execution on Docker, Slurm, or cloud backends.

Updated Mar 23, 2026
Search Tags:#benchmarking#llm evaluation#docker#slurm#nvidia#mmlu#nemo
One-click install
npx skills add https://github.com/hanasho744/codex --skill nemo-evaluator-sdk-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-evaluator-sdk
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/orchestra-nemo-evaluator
Command: npx skills add https://github.com/hanasho744/codex --skill nemo-evaluator-sdk-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-evaluator-launcher, docker, and includes references (resource) components.

What problem does it solve? Running consistent, reproducible LLM benchmarks across many harnesses and infrastructure types is complex and error-prone. This Skill standardizes evaluation of language and vision-language models across 100+ benchmarks from 18+ harnesses with container-first execution on local Docker, Slurm HPC clusters, or Lepton cloud. ## Core Features & Use Cases - Multi-harness benchmarking: Run MMLU, HumanEval, GSM8K, GPQA, safety harnesses (Aegis, WildGuard, garak), and VLM tasks (OCRBench, ChartQA, MMMU) through one unified CLI. - Multi-backend execution: Execute evaluations locally via Docker, on Slurm HPC clusters with vLLM/SGLang/NIM deployment, or on Lepton cloud, with job status tracking and kill controls. - Result export and comparison: Export results to MLflow, Weights & Biases, or local JSON to compare multiple models on identical tasks. - Use Case: Benchmark Llama 3.1 8B and Mistral 7B on MMLU Pro, GSM8K, and IFEval against NVIDIA's hosted API, then export both runs to MLflow for side-by-side comparison. ## Quick Start Ask the AI to evaluate a model on the ifeval benchmark using the nemo-evaluator-launcher CLI with a config pointing to your OpenAI-compatible endpoint and NGC API key.

Frequently Asked Questions about nemo-evaluator-sdk

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an LLM on MMLU or GSM8K with NeMo Evaluator?▼

Create a Hydra YAML config specifying your model endpoint and list tasks like mmlu_pro or gsm8k_cot_instruct under evaluation.tasks, then run nemo-evaluator-launcher run with your config directory. Results are written to the configured output directory.

How do I run LLM benchmarks on a Slurm HPC cluster?▼

Select the slurm execution backend in your config defaults and provide hostname, account, partition, and GPU settings. The launcher submits jobs via SSH and sbatch, and you can monitor status with nemo-evaluator-launcher status using the invocation ID.

NeMo Evaluator vs lm-evaluation-harness: which should I use?▼

NeMo Evaluator suits large-scale reproducible evaluation across 18+ harnesses with Slurm or cloud execution and result export. lm-evaluation-harness is simpler for quick local runs when you only need its supported tasks.

Does NeMo Evaluator support vision-language model benchmarks?▼

Yes, set the api_endpoint type to vlm and use tasks from vlmevalkit such as ocrbench, chartqa, and mmmu. These evaluate OCR, chart understanding, and multimodal reasoning on compatible vision-language endpoints.

Why does my evaluation task fail with a missing environment variable error?▼

Tasks like gpqa_diamond and mmlu require HF_TOKEN, and judge-based tasks need JUDGE_API_KEY. Map them in the task's env_vars section so the launcher passes them into the evaluation container.

Can I add custom benchmarks to NeMo Evaluator?▼

Yes, define custom tasks with Framework Definition Files and package your framework in a custom container registered in mapping.toml. Programmatic Python metric plugins are not currently supported; scoring must come through the FDF output parser.

Related Skills

Explore More Software Engineering Skills→
davila7davila7
30.5k

nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks using containerized multi-backend execution on Docker, Slurm, or cloud.

Community
Advanced
tadod12tadod12

nemo-evaluator-sdk

Evaluate LLMs across 100+ benchmarks from 18+ harnesses.

Community
Intermediate
PeterHiroshiPeterHiroshi
3

bg-dispatch

Dispatch long-running coding agents to background tasks with resumable progress.

Community
Intermediate
tianhao909tianhao909
1

evaluating-code-models

Evaluate code generation models with pass@k metrics across multiple benchmarks.

Community
Advanced