skills.rest
faramirezs
faramirezsCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Configure vLLM for high-throughput LLM serving with OpenAI-compatible endpoints.

Updated Jun 15, 2026
Search Tags:#production#llm#quantization#inference#tensor-parallel#vllm#serving
One-click install
npx skills add https://github.com/faramirezs/talking_printer --skill serving-llms-vllm-faramirezs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/faramirezs/talking_printer/tree/main/hermes-agent/skills/mlops/inference/vllm
Command: npx skills add https://github.com/faramirezs/talking_printer --skill serving-llms-vllm-faramirezs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

vLLM-based serving enables production-grade, high-throughput LLM inference with OpenAI-compatible endpoints, addressing the need for scalable, low-latency model serving.

Core Features & Use Cases

  • High-throughput serving using PagedAttention and continuous batching for large models.
  • OpenAI-compatible endpoints and easy SDK integration for rapid development.
  • Quantization support (AWQ, GPTQ, FP8) and tensor parallelism to fit large models on multi-GPU setups.
  • Typical use case: deploy a chat API that handles 100+ requests per second with predictable latency.

Quick Start

Start a local vLLM server configured with your chosen model and expose a v1-compatible endpoint for clients.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve large language models in production with high throughput?▼

Serve large language models in production by deploying vLLM to enable high-throughput inference using PagedAttention and continuous batching, achieving predictable latency for API workloads.

Can I use tensor parallelism to fit large models across multiple GPUs?▼

Yes, tensor parallelism fits large models across multi-GPU setups by configuring the --tensor-parallel-size flag, distributing model layers to maximize available compute resources during inference.

How do I set up an OpenAI-compatible endpoint for my LLM API?▼

Set up an OpenAI-compatible endpoint by starting a vLLM server with your chosen model, exposing a v1-compatible endpoint that allows rapid SDK integration for client applications.

Does vLLM support quantization methods like AWQ, GPTQ, and FP8?▼

vLLM supports quantization using AWQ, GPTQ, and FP8 methods by applying the optional --quantization flag, reducing memory footprint while maintaining inference quality for large models.

What are the prerequisites for deploying vLLM on Linux or macOS?▼

Deploying vLLM on Linux or macOS requires a Python environment with vllm installed, access to a suitable model, and configuration options like --gpu-memory-utilization for production serving.

How does continuous batching improve LLM serving latency?▼

Continuous batching improves LLM serving latency by dynamically processing incoming requests together using PagedAttention, enabling chat APIs to handle 100+ requests per second with predictable response times.

Related Skills

Explore More Software Engineering Skills→
hsliuustc0106hsliuustc0106
84

vllm-omni-setup

Install and configure vLLM-Omni with Python environments and smoke tests.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Deploy and optimize LLM inference servers with vLLM using OpenAI-compatible APIs and quantization.

Community
Advanced
chicanoandres702chicanoandres702

grafana-dashboards

Design and deploy production Grafana dashboards for system and application metrics.

Community
Advanced
icyzhicyzh
1

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints.

Community
Advanced