skills.rest
AlexiosBluffMara
AlexiosBluffMaraCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve large language models via vLLM with OpenAI-compatible endpoints and quantization.

2|Updated Apr 25, 2026
Search Tags:#LLM#throughput#quantization#OpenAI-compatible#inference-serving#vllm#tensor-parallelism
One-click install
npx skills add https://github.com/AlexiosBluffMara/mercury --skill serving-llms-vllm-alexiosbluffmara
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/AlexiosBluffMara/mercury/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/AlexiosBluffMara/mercury --skill serving-llms-vllm-alexiosbluffmara

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers.

What problem does it solve?

This skill provides production-grade serving of large language models using vLLM, enabling high throughput with OpenAI-compatible endpoints and memory-efficient deployment via PagedAttention and continuous batching.

Core Features & Use Cases

  • High-throughput OpenAI-compatible API serving for LLMs
  • Memory-efficient deployment with quantization (GPTQ/AWQ/FP8) and tensor parallelism
  • Suitable for production APIs, batch inference, and multi-model deployments

Quick Start

Launch the vLLM server with your model to start serving OpenAI-compatible requests immediately.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with an OpenAI-compatible API using vLLM?▼

To serve an LLM with an OpenAI-compatible API using vLLM, launch the vLLM server with your selected model. This immediately provides high-throughput OpenAI-compatible endpoints for production requests.

What is the best way to run batch inference on memory-constrained GPUs?▼

The best way to run batch inference on memory-constrained GPUs is using vLLM with quantization formats like GPTQ, AWQ, or FP8. This reduces memory consumption while maintaining production-grade throughput.

How does PagedAttention and continuous batching improve LLM throughput?▼

PagedAttention and continuous batching improve LLM throughput by dynamically managing memory pages and grouping incoming requests. This mechanism significantly maximizes GPU utilization during inference serving.

Can I use tensor parallelism in vLLM for multi-GPU model deployment?▼

Yes, you can use tensor parallelism in vLLM for multi-GPU model deployment. It splits model tensors across multiple GPUs, enabling low-latency chat services and high-throughput inference for large models.

Does vLLM support GPTQ and AWQ quantization for inference serving?▼

Yes, vLLM supports GPTQ, AWQ, and FP8 quantization for inference serving. These formats enable memory-efficient deployment of large language models on hardware with limited VRAM.

What are the limitations of using vLLM for production LLM serving?▼

Limitations of using vLLM for production LLM serving include strict dependencies on torch and transformers. Users must ensure their GPU hardware supports the required tensor parallelism and quantization features.

Related Skills

Explore More Software Engineering Skills→
kaminocorpkaminocorp
1

serving-llms-vllm

Serve LLMs with high throughput using vLLM's PagedAttention and continuous batching.

Official
Advanced
projectedanxprojectedanx
1

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints and continuous batching.

Community
Advanced
HouseGarofaloHouseGarofalo
3

nginx-proxy

Configure Nginx reverse proxy for SSL termination, load balancing, caching, and rate limiting.

Official
Intermediate
aws-samplesaws-samples
18

vllm-serving-setup

Automate vLLM inference serving deployment on EKS Kubernetes clusters.

Official
Advanced