skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT compilation, quantization, and multi-GPU parallelism.

13.0k|930|Updated Nov 3, 2025
Search Tags:#quantization#fp8#model serving#tensor parallelism#llm inference#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill tensorrt-llm-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill tensorrt-llm-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Serving large language models in production with PyTorch is slow and expensive, often failing to meet throughput and latency targets on NVIDIA GPU infrastructure. ## Core Features & Use Cases - High-Performance Inference: Delivers up to 100x faster inference than PyTorch using in-flight batching, paged KV cache, Flash Attention, and CUDA graphs. - Quantization Support: Runs FP8, INT4 (AWQ/GPTQ), and FP4 quantized models for 2-4x speedups and major memory savings. - Multi-GPU & Production Serving: Scales models like Llama 3-405B across GPUs and nodes with tensor/pipeline/expert parallelism, and serves them via an OpenAI-compatible API with Prometheus metrics and Kubernetes deployment patterns. - Use Case: Deploy Llama 3-70B on 4x A100 GPUs with FP8 quantization and trtllm-serve to handle thousands of concurrent chat requests at 10,000+ tokens/sec. ## Quick Start Use the tensorrt-llm skill to deploy Llama 3-8B as an OpenAI-compatible inference server on my NVIDIA GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with TensorRT-LLM?▼

Use the trtllm-serve command with your HuggingFace model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B --tp_size 4 --max_batch_size 256. It automatically downloads and compiles the model, then exposes an OpenAI-compatible API on port 8000.

TensorRT-LLM vs vLLM for LLM inference?▼

TensorRT-LLM delivers maximum throughput on NVIDIA GPUs through compilation and optimized kernels but requires more setup. vLLM offers a simpler Python-first API with PagedAttention and works on non-NVIDIA hardware like AMD GPUs.

Does TensorRT-LLM support FP8 quantization?▼

Yes, TensorRT-LLM supports FP8 quantization by setting dtype="fp8" when initializing the LLM, giving roughly 2x faster inference and 50% memory reduction on H100 GPUs with under 1% perplexity degradation. INT4 via AWQ or GPTQ is also supported.

How do I run a 70B model across multiple GPUs with TensorRT-LLM?▼

Set tensor_parallel_size to split the model across GPUs, for example tensor_parallel_size=4 for Llama 3-70B on 4x A100 80GB with FP8. For 405B models, combine tensor parallelism within nodes and pipeline parallelism across nodes with InfiniBand.

Why does TensorRT-LLM run out of memory during serving?▼

OOM occurs when batch size or token limits exceed available GPU memory. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to shard the model across more GPUs.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPU, Apple Silicon, or AMD GPUs, since it requires NVIDIA hardware and CUDA. Use llama.cpp for CPU or edge deployment and vLLM for simpler setups or non-NVIDIA accelerators.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

ptq

Quantize pretrained models to nvfp4, fp8, or int4_awq checkpoints using ModelOpt PTQ.

Official
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
air-gappedair-gapped
5

aiperf

Benchmark latency, throughput, and telemetry across OpenAI-compatible inference servers.

Official
Advanced
GregsGreyCodeGregsGreyCode
6

gguf-quantization

Quantize GGUF models for CPU inference on Apple Silicon and consumer hardware.

Community
Advanced