skills.rest
superfhp
superfhpCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy and optimize LLM inference servers with vLLM using OpenAI-compatible APIs and quantization.

Updated May 13, 2026
Search Tags:#quantization#inference#vllm#openai api#tensor parallelism#llm serving#gpu deployment
One-click install
npx skills add https://github.com/superfhp/lumi-agent-body --skill serving-llms-vllm-superfhp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/superfhp/lumi-agent-body/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/superfhp/lumi-agent-body --skill serving-llms-vllm-superfhp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving large language models in production is slow and memory-hungry with standard transformers. This Skill guides you through deploying high-throughput LLM APIs with vLLM, achieving up to 24x higher throughput via PagedAttention and continuous batching while fitting large models into limited GPU memory. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with monitoring, prefix caching, and Docker/Kubernetes configurations. - Offline Batch Inference: Process large prompt datasets efficiently with automatic batching and tensor parallelism. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization. - Use Case: Deploy a Llama-3-8B chatbot API handling 100+ requests per second on one A100 GPU, with Prometheus metrics tracking TTFT and throughput. ## Quick Start Use the vLLM skill to deploy a Llama-3-8B-Instruct model as an OpenAI-compatible API server on port 8000.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run 'vllm serve meta-llama/Llama-3-8B-Instruct' to launch an OpenAI-compatible server on port 8000. Query it using the OpenAI Python SDK with base_url set to http://localhost:8000/v1 and any API key value.

vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?▼

vLLM offers high throughput with easy setup and multi-platform support. TensorRT-LLM is about 12% faster but NVIDIA-only with complex setup. llama.cpp targets CPU and edge inference for single-user scenarios.

How do I fit a 70B model on a single 40GB GPU?▼

Use AWQ quantization with a pre-quantized model like TheBloke/Llama-2-70B-AWQ and launch with --quantization awq. This reduces memory from 140GB to about 35GB with under 1% accuracy loss.

Why does vLLM run out of memory during model loading?▼

The model plus KV cache exceeds available VRAM. Reduce --gpu-memory-utilization to 0.7, lower --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompts and chunked prefill with --enable-chunked-prefill for long prompts. Also reduce concurrent sequences if latency matters more than throughput.

Does vLLM support multiple GPUs and distributed serving?▼

Yes, use --tensor-parallel-size with a power-of-2 GPU count for single-node parallelism. For multi-node serving of very large models, combine tensor parallelism with --pipeline-parallel-size and configure MASTER_ADDR, RANK, and WORLD_SIZE environment variables.

Related Skills

Explore More Software Engineering Skills→
richardnguyen0715richardnguyen0715

peft-fine-tuning

Fine-tune LLMs with LoRA, QLoRA, and other PEFT methods on limited GPU memory.

Community
Advanced
masermediagroup-stackmasermediagroup-stack
1

model-serving

Deploy LLM and ML models for production inference with vLLM, BentoML, and streaming APIs.

Community
Advanced
jvdominojvdomino

domino-model-endpoints

Deploy and monitor model endpoints in Domino with Grafana and NVIDIA Triton support.

Community
Advanced
CUexterCUexter
2

llama-cpp

Run CPU-based LLM inference with llama.cpp on non-NVIDIA hardware.

Community
Advanced