skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

serving-llms-vllm

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.

13.0k|930|Updated Nov 3, 2025
Search Tags:#quantization#inference#vllm#pagedattention#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill serving-llms-vllm-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/12-inference-serving/vllm
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill serving-llms-vllm-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Deploying large language models in production often hits GPU memory limits, low throughput, and high latency. This Skill guides you through serving LLMs efficiently with vLLM, achieving up to 24x higher throughput than standard transformers through PagedAttention and continuous batching. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with monitoring, prefix caching, and Docker/Kubernetes deployment patterns. - Offline Batch Inference: Process large prompt datasets efficiently with automatic internal batching and tensor parallelism. - Quantized Model Serving: Fit 70B models on a single 40GB GPU using AWQ, GPTQ, or FP8 quantization with minimal accuracy loss. - Use Case: You need to deploy Llama-3-8B as a production chatbot API handling 100+ requests per second. Use this Skill to configure the server, enable prefix caching, set up Prometheus metrics, and verify TTFT stays under 500ms. ## Quick Start Use the vLLM skill to deploy meta-llama/Llama-3-8B-Instruct as an OpenAI-compatible API server on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run vllm serve with your model name to launch an OpenAI-compatible server on port 8000. Query it using the OpenAI Python SDK by setting base_url to http://localhost:8000/v1 and api_key to EMPTY.

vLLM vs TensorRT-LLM for LLM inference serving?▼

TensorRT-LLM is about 12% faster on throughput but is NVIDIA-only with high setup complexity. vLLM offers much easier setup, multi-platform support including AMD and Intel, and more quantization options like AWQ and GPTQ.

How do I fit a 70B model on a single 40GB GPU?▼

Use AWQ quantization with a pre-quantized model like TheBloke/Llama-2-70B-AWQ and launch with the --quantization awq flag. This reduces memory from 140GB to about 35GB with under 1% accuracy loss.

Why does vLLM throw out of memory errors during model loading?▼

OOM occurs when the model plus KV cache exceeds available VRAM. Reduce --gpu-memory-utilization to 0.7, lower --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompts, and use --enable-chunked-prefill for long prompts. Reducing concurrent sequences with --max-num-seqs also prioritizes latency over throughput.

When should I use llama.cpp instead of vLLM?▼

Use llama.cpp for CPU or edge inference and single-user scenarios. vLLM is designed for production GPU serving with high throughput, multi-user applications, and OpenAI-compatible endpoints handling 100+ requests per second.

Related Skills

Explore More Software Engineering Skills→
jayll1303jayll1303
17

model-quantization

Quantize large language models with GGUF, GPTQ, AWQ, and bitsandbytes.

Community
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
ArogyaReddyArogyaReddy

llm-evaluation

Automates LLM output evaluation with metrics, human feedback, and benchmarking.

Community
Advanced
jacardljacardl

llama-cpp

Provide CPU-based LLM inference for non-NVIDIA hardware using llama.cpp.

Community
Advanced