skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.

1|Updated Mar 5, 2026
Search Tags:#quantization#inference#vllm#pagedattention#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill serving-llms-vllm-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Clay-HHK/claude-skills/tree/main/vllm
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill serving-llms-vllm-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Deploying large language models in production often means choosing between slow inference, excessive GPU memory consumption, and complex serving infrastructure. This Skill guides you through serving LLMs with vLLM, achieving up to 24x higher throughput than standard transformers through PagedAttention and continuous batching. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with monitoring, prefix caching, and Docker/Kubernetes deployment patterns. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and tensor parallelism across multiple GPUs. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: You need to serve Llama-3-70B to hundreds of concurrent users but only have 2 A100 GPUs. Use this Skill to deploy an AWQ-quantized model with tensor parallelism, prefix caching, and Prometheus metrics monitoring. ## Quick Start Use the vllm skill to deploy meta-llama/Llama-3-8B-Instruct as an OpenAI-compatible API server on port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run 'vllm serve MODEL_NAME' to launch an OpenAI-compatible server on port 8000. Query it using the standard OpenAI Python SDK by setting base_url to http://localhost:8000/v1 and api_key to EMPTY.

vLLM vs TensorRT-LLM vs llama.cpp for inference serving?▼

Use vLLM for production APIs needing high throughput and multi-platform support. Choose TensorRT-LLM for maximum NVIDIA-only performance, llama.cpp for CPU/edge single-user inference, and HuggingFace transformers for research prototyping.

How do I serve a 70B model on limited GPU memory?▼

Use AWQ quantization to fit a 70B model in roughly 35GB VRAM on a single 40GB GPU. Launch with 'vllm serve TheBloke/Llama-2-70B-AWQ --quantization awq --gpu-memory-utilization 0.95'.

Why does vLLM throw out of memory errors during model loading?▼

OOM occurs when model weights plus KV cache exceed available VRAM. Reduce --gpu-memory-utilization to 0.7, lower --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompt prefixes, and use --enable-chunked-prefill for long prompts. Reducing --max-num-seqs also prioritizes latency over throughput.

Does vLLM support multi-GPU and multi-node serving?▼

Yes, vLLM supports tensor parallelism across multiple GPUs with --tensor-parallel-size (use powers of 2) and pipeline parallelism across nodes with --pipeline-parallel-size. Multi-node setups require matching MASTER_ADDR, MASTER_PORT, and WORLD_SIZE environment variables.

Related Skills

Explore More Software Engineering Skills→
box755box755

gptq

Apply GPTQ post-training quantization to compress LLM weights into 4-bit representations.

Community
Advanced
jvdominojvdomino

domino-model-endpoints

Deploy and monitor model endpoints in Domino with Grafana and NVIDIA Triton support.

Community
Advanced
air-gappedair-gapped
5

aiperf

Benchmark latency, throughput, and telemetry across OpenAI-compatible inference servers.

Official
Advanced
RedWoodOGRedWoodOG
174

llama-cpp

Run CPU-based LLM inference with llama.cpp using GGUF quantization.

Community
Intermediate