skills.rest
Nzettodess
NzettodessCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT compilation, quantization, and multi-GPU parallelism.

2|Updated Jan 26, 2026
Search Tags:#quantization#fp8#model serving#llm inference#tensorrt-llm#nvidia gpu#multi-gpu
One-click install
npx skills add https://github.com/Nzettodess/Awesome-Agent-Skills --skill tensorrt-llm-nzettodess
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Nzettodess/Awesome-Agent-Skills/tree/main/Skills/Agentic%20AI%20Development/AI-research-SKILLs-1.1.0/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/Nzettodess/Awesome-Agent-Skills --skill tensorrt-llm-nzettodess

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Serving large language models in production with PyTorch is slow and expensive, often failing to meet throughput and latency targets on NVIDIA GPU infrastructure. ## Core Features & Use Cases - High-Performance Inference: Achieves up to 100x faster inference than PyTorch using TensorRT compilation, in-flight batching, paged KV cache, and CUDA graphs. - Quantization Support: Runs FP8, INT4 AWQ, and INT4 GPTQ quantized models for 2-4x speedups and 50-75% memory reduction. - Multi-GPU Scaling: Supports tensor, pipeline, and expert parallelism across A100/H100 GPUs and multi-node clusters for models up to 405B parameters. - Use Case: Deploy Llama 3-70B on 4 A100 GPUs with FP8 quantization and serve it through an OpenAI-compatible API handling 10,000+ tokens per second with Prometheus monitoring. ## Quick Start Use the tensorrt-llm skill to deploy Llama 3-8B as an OpenAI-compatible server on my NVIDIA GPU with FP8 quantization.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve a Llama model with TensorRT-LLM?▼

Use the trtllm-serve command with your HuggingFace model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B with flags for tp_size and max_batch_size. It starts an OpenAI-compatible server on port 8000 that handles model download and compilation automatically.

TensorRT-LLM vs vLLM for LLM inference serving?▼

TensorRT-LLM delivers maximum throughput and lowest latency on NVIDIA GPUs through TensorRT compilation and FP8 quantization. vLLM offers a simpler Python-first setup with PagedAttention and works on non-NVIDIA hardware, making it better for quick prototyping.

Does TensorRT-LLM support FP8 and INT4 quantization?▼

Yes, TensorRT-LLM supports FP8 quantization for roughly 2x speedup and 50% memory reduction on H100 GPUs, plus INT4 via AWQ or GPTQ calibration for 3-4x speedups. Set the dtype parameter to fp8, int4_awq, or int4_gptq when initializing the LLM.

How do I run a 70B model across multiple GPUs with TensorRT-LLM?▼

Set tensor_parallel_size to split the model across GPUs, such as tp_size 4 for Llama 3-70B on four A100 GPUs with NVLink. For models above 320GB like Llama 3-405B, combine tensor parallelism within nodes and pipeline parallelism across nodes.

Why does TensorRT-LLM run out of GPU memory?▼

OOM errors occur when batch size or token limits exceed available memory after model loading. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to distribute the model across more GPUs.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPUs, Apple Silicon, or AMD GPUs, where llama.cpp or other runtimes are appropriate. It is also unnecessary for simple prototypes where vLLM's easier setup outweighs the compilation overhead.

Related Skills

Explore More Software Engineering Skills→
AlexKonceptAlexKoncept

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs using TensorRT-LLM with FP8/INT4 quantization.

Community
Advanced
allthingslinuxallthingslinux
1

prometheus-grafana

Deploy Prometheus and Grafana to monitor systems and services.

Official
Advanced
BBufBBuf
721

llm-serving-auto-benchmark

Benchmark LLM serving frameworks across SGLang, vLLM, and TensorRT-LLM.

Community
Advanced
JKhyroJKhyro

llama-cpp

Run local LLM inference with GGUF models on non-NVIDIA hardware.

Community
Advanced