skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

sglang

Serve LLMs with RadixAttention prefix caching and constrained structured generation.

13.0k|930|Updated Nov 3, 2025
Search Tags:#inference#structured generation#prefix caching#sglang#constrained decoding#llm serving#radixattention
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill sglang-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sglang
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/12-inference-serving/sglang
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill sglang-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sglang, torch, transformers, and includes references (resource) components.

What problem does it solve? Serving LLMs in production wastes compute when every request recomputes shared prefixes like system prompts, few-shot examples, and tool definitions, and generating valid JSON or regex-constrained output from models is unreliable without constrained decoding. ## Core Features & Use Cases - RadixAttention Prefix Caching: Automatically caches and reuses KV cache for shared prefixes across requests, delivering 5-10x speedups for agent workflows and few-shot prompting. - Structured Generation: Enforce JSON schemas, regex patterns, and EBNF grammars during decoding so outputs always match the required format. - Production Deployment: OpenAI-compatible API, tensor parallelism, FP8/AWQ/GPTQ quantization, Prometheus metrics, and Kubernetes deployment guidance. - Use Case: Build an agent that handles hundreds of user queries against the same system prompt and tool definitions; SGLang reuses the cached prefix so each subsequent request only computes the new tokens. ## Quick Start Install sglang with pip, launch a server for a Llama model on port 30000, and send a chat completion request to the OpenAI-compatible endpoint.

Frequently Asked Questions about sglang

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate structured JSON output from an LLM?▼

Use SGLang's constrained decoding by passing a json_schema or regex parameter to sgl.gen inside an @sgl.function program. The decoder enforces the schema token by token, so the output always matches the required structure.

SGLang vs vLLM for LLM serving, which should I use?▼

SGLang is faster when requests share prefixes, such as agent system prompts or few-shot examples, thanks to RadixAttention caching, and it supports constrained decoding natively. vLLM is a mature choice for simple unstructured text generation without prefix reuse.

What is RadixAttention and how does it speed up inference?▼

RadixAttention stores processed tokens in a radix tree and reuses the KV cache for shared prefixes across requests. Agent workloads with repeated system prompts run about 5x faster, and few-shot prompting can see up to 10x speedup.

Does SGLang support multi-GPU deployment and quantization?▼

Yes, SGLang supports tensor parallelism via the --tp flag for multi-GPU serving, and quantization with FP8 on H100 or AWQ and GPTQ INT4 formats. It runs on NVIDIA A100 and H100 as well as AMD MI300 and MI250 GPUs.

Why is my SGLang cache hit rate low?▼

Low hit rates occur when prompts lack a common structure, contain dynamic content like timestamps in the prefix, or the cache is too small and evicting entries. Restructure prompts so shared content comes first and increase --max-radix-cache-len.

When should I not use SGLang?▼

Avoid SGLang for simple text generation without shared prefixes, where vLLM is sufficient, or when you need maximum single-request latency with FP8 or INT4 on NVIDIA-only hardware, where TensorRT-LLM fits better.

Related Skills

Explore More Software Engineering Skills→
Chris-Chai-MinjaeChris-Chai-Minjae

llama-cpp

Discover GGUF models on Hugging Face Hub and generate llama-server or llama-cli commands.

Community
Advanced
api7api7
1

a6-plugin-prometheus

Expose APISIX metrics in Prometheus format for monitoring and dashboards.

Official
Intermediate
Supporter09Supporter09

sglang

Generate structured, prefix-cached LLM outputs for serving workloads.

Community
Advanced
nadicodeainadicodeai

tensorrt-llm

Deploy high-throughput LLM inference on NVIDIA GPUs with tensorrt-llm.

Official
Advanced