skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs with TensorRT compilation, quantization, and multi-GPU parallelism.

1|Updated Mar 5, 2026
Search Tags:#quantization#fp8#model serving#tensor parallelism#llm inference#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill tensorrt-llm-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Clay-HHK/claude-skills/tree/main/tensorrt-llm
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill tensorrt-llm-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Deploying large language models in production often suffers from high latency, low throughput, and excessive GPU memory consumption when using standard PyTorch inference. This Skill guides you through compiling and serving LLMs with NVIDIA TensorRT-LLM to achieve dramatically faster inference on NVIDIA hardware. ## Core Features & Use Cases - Quantized Inference: Run models in FP8, INT4 (AWQ/GPTQ), or FP4 to cut memory usage by 50-75% and multiply throughput. - Production Serving: Launch an OpenAI-compatible API server with trtllm-serve, including in-flight batching, streaming, Prometheus metrics, and Kubernetes deployment patterns. - Multi-GPU Scaling: Configure tensor, pipeline, and expert parallelism to serve models from 8B up to 405B parameters across multiple GPUs and nodes. - Use Case: Serve Llama 3-70B on 4 A100 GPUs with FP8 quantization and tensor parallelism, exposing an OpenAI-compatible endpoint that handles hundreds of concurrent requests with in-flight batching. ## Quick Start Use the tensorrt-llm skill to serve Llama 3-8B with an OpenAI-compatible API on my NVIDIA GPU.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with TensorRT-LLM?▼

Use the trtllm-serve command with your HuggingFace model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B with flags for tp_size and max_batch_size. It starts an OpenAI-compatible server on port 8000 that handles model download and compilation automatically.

TensorRT-LLM vs vLLM for LLM inference?▼

TensorRT-LLM delivers maximum throughput and lowest latency on NVIDIA GPUs through compilation, FP8 quantization, and CUDA graphs, but requires NVIDIA hardware and more setup. vLLM offers a simpler Python-first API with PagedAttention and works on non-NVIDIA hardware.

Does TensorRT-LLM support FP8 quantization?▼

Yes, TensorRT-LLM supports FP8 quantization by passing dtype="fp8" when initializing the LLM class. On H100 GPUs this provides roughly 2x faster inference and 50% memory reduction with minimal accuracy loss, and INT4 AWQ or GPTQ is also available.

How do I run a 70B model across multiple GPUs with TensorRT-LLM?▼

Set tensor_parallel_size when creating the LLM instance, for example tensor_parallel_size=4 for Llama 3-70B on four A100 GPUs with FP8. For very large models like 405B, combine tensor parallelism within nodes and pipeline parallelism across nodes.

Why does TensorRT-LLM run out of GPU memory?▼

OOM errors occur when batch size, max_num_tokens, or model size exceeds available VRAM. Reduce max_batch_size, enable FP8 or INT4 quantization, increase tensor_parallel_size to shard the model, or enable chunked context for long prompts.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPUs, Apple Silicon, or AMD GPUs, since it requires NVIDIA hardware. Use llama.cpp for CPU or edge deployment with GGUF models, and vLLM when you need simpler setup without TensorRT compilation.

Related Skills

Explore More Software Engineering Skills→
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
api7api7
1

a6-plugin-prometheus

Expose APISIX metrics in Prometheus format for monitoring and dashboards.

Official
Intermediate
air-gappedair-gapped
5

aiperf

Benchmark latency, throughput, and telemetry across OpenAI-compatible inference servers.

Official
Advanced
JKhyroJKhyro

llama-cpp

Run local LLM inference with GGUF models on non-NVIDIA hardware.

Community
Advanced