What problem does it solve? Benchmarking Hugging Face Hub models locally requires choosing between evaluation frameworks (inspect-ai vs lighteval) and inference backends (vLLM, Transformers, accelerate), each with different flags, task formats, and failure modes. This Skill provides ready-to-run scripts and decision guidance so you can execute smoke tests and full evals on local hardware without assembling the toolchain yourself. ## Core Features & Use Cases - inspect-ai evaluations: Run tasks like mmlu, gsm8k, and humaneval via inference providers or local GPU with vLLM/Transformers backends. - lighteval evaluations: Execute leaderboard-style task strings (e.g., "leaderboard|mmlu|5") with vLLM or accelerate backends, including chat template support. - Backend fallback strategy: Automatically guides switching from vLLM to Transformers or accelerate when a model architecture is unsupported, plus OOM troubleshooting steps. - Use Case: You want to compare Llama-3.2-3B-Instruct against phi-2 on MMLU and GSM8K before deployment. Run a 20-sample smoke test locally with vLLM, fall back to accelerate for phi-2, then hand off larger runs to Hugging Face Jobs. ## Quick Start Ask the AI to run an MMLU smoke test on meta-llama/Llama-3.2-1B locally with a limit of 20 samples using the inspect-ai vLLM script.
