skills.rest
shakfu
shakfuCOMMUNITY
Explore all skills by this author →
GitHub

serve-model

Launch and configure OpenAI-compatible model servers with MAX's max serve command.

Updated Sep 15, 2026
Search Tags:#quantization#model serving#openai api#llm deployment#inference server#max serve#custom architecture
One-click install
npx skills add https://github.com/shakfu/mdsp --skill serve-model-shakfu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serve-model
Source: https://github.com/shakfu/mdsp/tree/main/.claude/skills/serve-model
Command: npx skills add https://github.com/shakfu/mdsp --skill serve-model-shakfu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface_hub, and includes scripts (resource) and references (resource) components.

What problem does it solve? Serving a model with MAX involves choosing the right environment setup, device targets, quantization encodings, and serve flags, and a wrong choice turns a working serve into a confusing startup failure. This Skill takes you from no environment to a running OpenAI-compatible endpoint, with flag choices derived from the model package and checkpoint instead of guesswork. ## Core Features & Use Cases - Environment Setup: Install MAX nightly via pixi (conda channel) or uv (wheel index) and verify the max serve command is available. - Custom Architecture Serving: Point --custom-architectures at a package, derive flags from arch.py and config.json, and run the bundled inspector script to get a ready-to-run command with explanatory notes. - Launch and Verification: Start the server detached on remote hosts, watch the log for readiness or crash markers, and confirm with a real chat completion request. - Troubleshooting: Match startup failures (name mismatches, encoding errors, OOM, port collisions) to one-line fixes via the troubleshooting reference. - Use Case: You have a ported Llama-variant checkpoint and a custom architecture package. Run the suggest_serve_command.py script to get the correct flags, launch with setsid on a remote GPU box, wait for the readiness marker, and verify coherent output. ## Quick Start Ask the AI to serve your Hugging Face model or local checkpoint with MAX, mentioning whether it uses a custom architecture package.

Frequently Asked Questions about serve-model

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve a model with MAX max serve?▼

Run max serve --model <hf-repo-or-path> after installing MAX nightly via pixi or uv. MAX auto-detects device, dtype, and sequence length, so the minimal command works for most built-in architectures on a single GPU.

How do I serve a custom architecture with MAX?▼

Pass --custom-architectures pointing at the package directory whose __init__.py exports an ARCHITECTURES list. The package's arch.py name field must exactly match the checkpoint's config.json architectures[0], and your quantization-encoding must be in supported_encodings.

Does MAX serve support CPU inference?▼

Yes, MAX serves on CPU when you pass --devices cpu, which is practical for float32 checkpoints. GPU is the default, and encodings like float8, float4, and gptq are GPU-only.

Why does max serve fail with not found in registry?▼

This error means the checkpoint's config.json architectures[0] does not match any registered SupportedArchitecture name. For a custom arch, make the name field in arch.py exactly equal to architectures[0]; otherwise the model needs a bring-up port first.

Why does max serve hang during startup?▼

Large models compile on first launch, printing Still compiling model heartbeats with an advancing elapsed counter, which is normal. Only treat it as stuck if the counter freezes or the log stops updating while the process is alive.

When should I not use max serve for a model?▼

Do not use it when the model architecture is not implemented in MAX yet, meaning there is no working arch.py, graph, and weight adapters. That is a model bring-up task requiring a porting workflow, not a serving one.

Related Skills

Explore More Software Engineering Skills→
automatedigitalautomatedigital
1

peft-fine-tuning

Fine-tune large language models by training less than 1% of weights via LoRA and QLoRA.

Official
Advanced
copilot-ldcopilot-ld
1

libllm

Expose an HTTP LLM API client for OpenAI-compatible chat completions and embeddings.

Official
Advanced
nikhillinitnikhillinit

phoenix-xirr-fees-validator

Validate XIRR and fee calculations against Excel parity and truth cases.

Community
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced