skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT with quantization and multi-GPU parallelism.

1|Updated Mar 12, 2026
Search Tags:#quantization#fp8#model serving#tensor parallelism#llm inference#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill tensorrt-llm-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill tensorrt-llm-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Deploying large language models in production often suffers from high latency, low throughput, and excessive GPU memory consumption when using standard PyTorch inference. This Skill provides guidance for compiling and serving LLMs with NVIDIA TensorRT-LLM to achieve dramatically faster inference on NVIDIA hardware. ## Core Features & Use Cases - Quantized Inference: Run models in FP8, INT4 (AWQ/GPTQ), or FP4 to cut memory usage by 50-75% and multiply throughput. - Multi-GPU Scaling: Configure tensor, pipeline, and expert parallelism to serve models from 8B to 405B parameters across single nodes or multi-node clusters. - Production Serving: Deploy an OpenAI-compatible API server with in-flight batching, Prometheus metrics, health checks, Kubernetes manifests, and autoscaling. - Use Case: Serve Llama 3-70B on 4 A100 GPUs with FP8 quantization and tensor parallelism, exposing an OpenAI-compatible endpoint that handles hundreds of concurrent requests with in-flight batching. ## Quick Start Set up a TensorRT-LLM server to serve Llama 3-8B with FP8 quantization on my available NVIDIA GPUs and show me how to send a chat completion request to it.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with TensorRT-LLM?▼

Use the trtllm-serve command with your HuggingFace model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B with flags for tensor parallelism and batch size. It starts an OpenAI-compatible server on port 8000 that accepts chat completion requests.

TensorRT-LLM vs vLLM for LLM inference?▼

TensorRT-LLM delivers maximum throughput on NVIDIA GPUs through compilation, quantization, and optimized kernels, but requires NVIDIA hardware and more setup. vLLM offers a simpler Python-first API with PagedAttention and works on non-NVIDIA hardware.

Does TensorRT-LLM support FP8 quantization?▼

Yes, TensorRT-LLM supports FP8 quantization by setting dtype to fp8 when initializing the LLM, delivering roughly 2x faster inference and 50% memory reduction on H100 GPUs. It also supports INT4 via AWQ and GPTQ calibration for greater compression.

Can TensorRT-LLM run models across multiple GPUs?▼

Yes, it supports tensor parallelism to split layers across GPUs on one node, pipeline parallelism for multi-node distribution, and expert parallelism for MoE models. For example, Llama 3-405B can run on 16 H100 GPUs with TP=8 and PP=2.

Why does TensorRT-LLM run out of memory during serving?▼

OOM errors occur when batch size or token limits exceed available GPU memory. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to shard the model across more GPUs.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPUs, Apple Silicon, or AMD GPUs, since it requires NVIDIA hardware. Use llama.cpp for CPU or edge deployment with GGUF models, or vLLM when you need simpler setup without TensorRT compilation.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

ptq

Quantize pretrained models to nvfp4, fp8, or int4_awq checkpoints using ModelOpt PTQ.

Official
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
cryptopaficryptopafi

serving-llms-vllm

Serve large language models with high-throughput inference for OpenAI-compatible endpoints.

Community
Advanced
JKhyroJKhyro

llama-cpp

Run local LLM inference with GGUF models on non-NVIDIA hardware.

Community
Advanced