Benchmarks text models served by MAX against standard datasets like GSM8K and MMLU.
npx skills add https://github.com/shakfu/mdsp --skill eval-model-shakfu
Please help me install this Agent Skill. Skill: eval-model Source: https://github.com/shakfu/mdsp/tree/main/.claude/skills/eval-model Command: npx skills add https://github.com/shakfu/mdsp --skill eval-model-shakfu
High-intent search queries and answers about installing and using this skill.
Install the evaluator environment with pixi install, serve the model with max serve, then run pixi run eval with --model and --tasks. Generation tasks like GSM8K use chat completions; multiple-choice tasks like MMLU require the server started with --enable-echo.
Multiple-choice tasks such as MMLU and HellaSwag score by prompt log probabilities, which MAX only returns when the server starts with --enable-echo. Echo cannot be toggled on a running server, so restart it before evaluating those tasks.
Look up the reference metric and shot count on the model card, then pass it as a one-sided floor with --target, for example --target "gsm8k:0.84". The run exits with code 2 if accuracy falls below the target; allow for sampling noise at the default 200-sample limit.
The served model's generation_config.json sets a non-default repetition_penalty (common in Qwen2.5), and MAX auto-enables a penalty sampling path that is shape-incompatible with echo. Use a model that leaves repetition_penalty unset, or restrict the run to generation tasks.
Yes, pass --reasoning-effort with low, medium, high, or xhigh when the model's chat template accepts reasoning_effort. The evaluator sends it via chat_template_kwargs and the preflight request fails clearly if the endpoint rejects the field.
Use --direct-http only for connectivity and parsing checks. Its bundled scorer is 0-shot with a custom prompt, so scores run far below few-shot chain-of-thought model-card numbers and must not be compared with published benchmarks.
rbkayzGreet users and echo back their provided text input.
jadecliCompute completeness, accuracy, and freshness scores to gate crawl output quality.
jimmymalhanEmit periodic heartbeat signals and detect stalls in long-running tasks.
lucasfthBenchmark and compare LLMs across 60+ tasks with lm-evaluation-harness.
