skills.rest
AlexKoncept
AlexKonceptCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs using TensorRT-LLM with FP8/INT4 quantization.

Updated Aug 23, 2026
Search Tags:#quantization#fp8#llm-inference#tensorrt#int4#multi-gpu
One-click install
npx skills add https://github.com/AlexKoncept/omnia-hub --skill tensorrt-llm-alexkoncept
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/AlexKoncept/omnia-hub/tree/main/HERMES/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/AlexKoncept/omnia-hub --skill tensorrt-llm-alexkoncept

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference with TensorRT-LLM to deliver maximum throughput and minimum latency on NVIDIA GPUs in production deployments, supporting quantization and multi-GPU scaling.

Core Features & Use Cases

  • Quantization support: FP8 and INT4 for faster inference and reduced memory.
  • In-flight batching and CUDA graphs for higher throughput and stable latency.
  • Tensor and pipeline parallelism, multi-node deployment for very large models.

Quick Start

Install prerequisites and launch the server with your model to begin production-grade inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I speed up LLM inference on NVIDIA GPUs in production?▼

Optimize LLM inference latency on NVIDIA GPUs by applying TensorRT-LLM with in-flight batching, CUDA graphs, and FP8 or INT4 quantization to maximize throughput and minimize latency in production deployments.

What's the best way to reduce memory usage during LLM inference?▼

Reduce memory usage during LLM inference by applying FP8 and INT4 quantization via TensorRT-LLM. This achieves faster computation and reduced memory footprint while maintaining stable latency for production deployments.

Does TensorRT-LLM support multi-node deployment for very large models?▼

TensorRT-LLM supports multi-node deployment for very large models by utilizing tensor and pipeline parallelism. It scales across multi-GPU clusters enabled by NVLink and InfiniBand to handle massive inference workloads.

What prerequisites do I need to run TensorRT-LLM for inference?▼

To run TensorRT-LLM for inference, you need a CUDA-compatible environment with the CUDA toolkit, NVIDIA drivers, and a compatible PyTorch version. These prerequisites satisfy the requirements for serving and benchmarking.

How does in-flight batching improve LLM inference throughput?▼

In-flight batching improves LLM inference throughput by dynamically scheduling incoming requests alongside active computations. Combined with CUDA graphs in TensorRT-LLM, it ensures higher throughput and stable latency.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
zerwizzerwiz

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM and OpenAI-compatible endpoints.

Community
Advanced
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced