skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

sglang

Serve LLMs with RadixAttention prefix caching and constrained structured generation.

1|Updated Mar 5, 2026
Search Tags:#json schema#inference#structured generation#prefix caching#sglang#llm serving#radixattention
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill sglang-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sglang
Source: https://github.com/Clay-HHK/claude-skills/tree/main/sglang
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill sglang-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sglang, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving LLMs in production wastes compute when every request recomputes shared prefixes like system prompts, few-shot examples, and tool definitions, and generating valid JSON or regex-constrained output from models is unreliable without constrained decoding. ## Core Features & Use Cases - RadixAttention Prefix Caching: Automatically caches and reuses KV cache for shared prefixes across requests, delivering 5-10x speedups for agent workflows and few-shot prompting. - Structured Generation: Constrain outputs with JSON schemas, regex patterns, or EBNF grammars for guaranteed-valid structured data extraction. - Production Deployment: OpenAI-compatible API server with tensor parallelism, FP8/AWQ/GPTQ quantization, Prometheus metrics, Kubernetes manifests, and autoscaling guidance. - Use Case: Build an agent that handles hundreds of queries against the same system prompt and tool definitions; SGLang caches the shared prefix so each new request only computes the unique user query, cutting latency from 1.8s to 0.35s. ## Quick Start Install sglang with pip, launch a server for a Llama model on port 30000, and send a chat completion request to the OpenAI-compatible endpoint.

Frequently Asked Questions about sglang

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I launch an SGLang inference server?▼

Run python -m sglang.launch_server with --model-path pointing to a HuggingFace model and --port for the endpoint. Add --tp 4 for tensor parallelism across multiple GPUs, then query the OpenAI-compatible API at /v1/chat/completions.

How do I generate JSON output with a schema in SGLang?▼

Pass a json_schema parameter to sgl.gen inside an @sgl.function decorated program, defining properties, types, and required fields. SGLang constrains decoding so the output always matches the schema, and regex or EBNF grammar constraints work similarly.

SGLang vs vLLM: which should I use for LLM serving?▼

Use SGLang when workloads share prefixes like system prompts or few-shot examples, since RadixAttention gives 5-10x speedups automatically, or when you need structured output. Use vLLM for simple unstructured generation without prefix sharing.

Does SGLang support quantization and multi-GPU deployment?▼

Yes, SGLang supports FP8 on H100 and AWQ/GPTQ INT4 quantization via the --quantization flag, plus tensor parallelism with --tp for multi-GPU serving. It runs on NVIDIA A100/H100/L4 and AMD MI300/MI250 hardware.

Why is my SGLang RadixAttention cache hit rate low?▼

Low hit rates occur when prompts lack common structure, contain dynamic content like timestamps in the prefix, or the cache is too small and evicting entries. Restructure prompts with shared content first and increase --max-radix-cache-len.

When should I not use SGLang?▼

Avoid SGLang for simple text generation without shared prefixes, where vLLM is more mature, or when you need maximum single-request latency with FP8/INT4 on NVIDIA-only deployments, where TensorRT-LLM fits better.

Related Skills

Explore More Software Engineering Skills→
agentic-inagentic-in
577

vLLM

Serve LLMs with OpenAI-compatible endpoints and high-throughput batching.

Official
Advanced
api7api7
1

a6-plugin-prometheus

Expose APISIX metrics in Prometheus format for monitoring and dashboards.

Official
Intermediate
suhanrsuhanr
2

ai-ml-engineer

Build, evaluate, and deploy machine-learning and generative-AI systems from data preparation through inference.

Community
Intermediate
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced