skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

serving-llms-vllm

Serve LLMs with high throughput using vLLM's PagedAttention and continuous batching.

1|Updated Mar 12, 2026
Search Tags:#quantization#inference#vllm#pagedattention#openai api#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill serving-llms-vllm-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/inference/vllm
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill serving-llms-vllm-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes references (resource) components.

What problem does it solve? Deploying large language models in production is bottlenecked by GPU memory waste, low throughput, and high latency when using standard inference frameworks. This Skill provides operational guidance for serving LLMs efficiently with vLLM, covering PagedAttention memory management, continuous batching, quantization, and production deployment patterns. ## Core Features & Use Cases - Production API Deployment: Launch OpenAI-compatible endpoints with tensor parallelism, prefix caching, and Prometheus metrics for monitoring TTFT and throughput. - Offline Batch Inference: Process large prompt datasets in a single call with automatic internal batching and configurable sampling parameters. - Quantized Model Serving: Fit 70B models into 40GB of VRAM using AWQ, GPTQ, or FP8 quantization with documented accuracy trade-offs. - Use Case: A team needs to serve Llama-3-8B to hundreds of concurrent chatbot users. Use this Skill to configure a vLLM server with prefix caching and continuous batching, achieving over 100 requests per second on a single A100. ## Quick Start Use the vLLM skill to deploy meta-llama/Llama-3-8B-Instruct as an OpenAI-compatible API server on port 8000 with prefix caching enabled.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM as an OpenAI-compatible API with vLLM?▼

Run vllm serve with your model name, for example vllm serve meta-llama/Llama-3-8B-Instruct, which starts an OpenAI-compatible server on port 8000. Query it with the OpenAI Python SDK by setting base_url to http://localhost:8000/v1.

vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?▼

vLLM offers high throughput with easy setup and multi-platform support, while TensorRT-LLM is roughly 12% faster but NVIDIA-only with complex setup. llama.cpp targets CPU and edge single-user inference, not production multi-user serving.

How do I fit a 70B model on a single 40GB GPU?▼

Use AWQ quantization with a pre-quantized model such as TheBloke/Llama-2-70B-AWQ and launch with --quantization awq. This reduces memory from 140GB to about 35GB with under 1% accuracy loss.

Why does vLLM run out of memory during model loading?▼

OOM occurs when the model plus KV cache exceeds available VRAM. Lower --gpu-memory-utilization to 0.7, reduce --max-model-len, enable AWQ quantization, or split the model across GPUs with --tensor-parallel-size.

How do I reduce time to first token in vLLM?▼

Enable prefix caching with --enable-prefix-caching for repeated prompt prefixes, and use --enable-chunked-prefill for long prompts. Reducing concurrent sequences with --max-num-seqs also prioritizes latency over throughput.

Does vLLM support monitoring and metrics in production?▼

Yes, vLLM exposes Prometheus metrics on a configurable port when launched with --enable-metrics. Key metrics include time to first token, running requests, and GPU cache usage, which integrate with Grafana dashboards.

Related Skills

Explore More Software Engineering Skills→
Orchestra-ResearchOrchestra-Research
11.5k

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight quantization.

Official
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
NVIDIANVIDIA
3.7k

evaluation

Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher.

Official
Advanced
Clay-HHKClay-HHK
1

llama-cpp

Runs quantized GGUF LLM inference on CPUs, Apple Silicon, and non-NVIDIA GPUs.

Community
Intermediate