Benchmark MAX model servers to measure throughput, latency, and GPU utilization.
npx skills add https://github.com/shakfu/mdsp --skill benchmark-model-shakfu
Please help me install this Agent Skill. Skill: benchmark-model Source: https://github.com/shakfu/mdsp/tree/main/.claude/skills/benchmark-model Command: npx skills add https://github.com/shakfu/mdsp --skill benchmark-model-shakfu
max benchmark against a max serve endpoint so you get trustworthy tokens/sec, TTFT, and TPOT figures instead of garbage or misleading results. ## Core Features & Use Cases - Workload Selection: Match the benchmark workload to your question, from single-request latency at concurrency 1 to concurrency and request-rate sweeps that reveal peak throughput and the latency knee. - Result Capture and GPU Stats: Save runs to JSON with --result-filename, stamp them with --metadata, and collect GPU utilization and peak memory with --collect-gpu-stats. - Troubleshooting Guidance: Diagnose connection failures, model-name mismatches, tokenizer-alias errors, chat-template 400s, and flat throughput caused by a server batch-size cap. - Use Case: You just deployed a model with max serve and need to know whether it can hold a TTFT p99 under 500 ms at production load. Run a concurrency sweep, read the highest concurrency that meets the SLA, and save the stamped JSON for comparison with future builds. ## Quick Start Ask the AI to benchmark the model currently served on localhost:8000 with a concurrency sweep and save the results to a JSON file.High-intent search queries and answers about installing and using this skill.
Run `max benchmark` against a live `max serve` endpoint, passing `--base-url`, the served model name as `--model`, a dataset such as `random`, and `--num-prompts`. Confirm `curl /v1/health` returns 200 first, and save output with `--result-filename`.
Run a concurrency sweep with `--max-concurrency 1,2,4,8,16,32` and enough prompts, typically 200 or more. The concurrency-1 point gives best-case TTFT and TPOT, while the peak output tokens/sec across the sweep is the throughput number.
Connection refused means nothing is serving or the host and port are wrong, so verify `curl /v1/health` returns 200. Model-not-found errors mean `--model` does not equal the server's `--served-model-name`; read the exact value from `curl /v1/models`.
Flat throughput across a concurrency sweep means the server's `--max-batch-size` caps real concurrency, so requests queue instead of batching. Re-serve with `--max-batch-size` at or above the top of your sweep and rerun the benchmark.
Yes, pass `--collect-gpu-stats` to report GPU utilization and peak memory. It works only on NVIDIA hardware and only when the benchmark runs on the same machine as the `max serve` server.
Do not use it when no server is running yet, since it is a client that requires a live endpoint. For kernel-level analysis of where inference time goes, use profiling instead, and for output correctness use a parity check rather than a benchmark.
cabeznoDeploy an OpenAI-compatible LLM inference server using vLLM.
diiviikk5Measure web page performance baselines and detect regressions in PR workflows.
fmh66Profile CUDA-C++ kernels with Nsight Compute metrics and classify bottlenecks.
wwwillottBenchmark language models on MMLU, GSM8K, HumanEval, and more via a unified harness.