tensorrt-llm
CommunityAccelerate LLM inference on NVIDIA GPUs.
Software Engineering#optimization#production serving#nvidia#gpu acceleration#llm inference#tensorrt-llm
AuthorAum08Desai
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill optimizes Large Language Model (LLM) inference for production environments on NVIDIA GPUs, drastically reducing latency and increasing throughput compared to standard frameworks.
Core Features & Use Cases
- High-Performance Inference: Achieve 10-100x faster inference speeds on NVIDIA A100/H100 GPUs.
- Production Deployment: Serve LLMs efficiently with features like in-flight batching, quantization (FP8/INT4), and multi-GPU scaling.
- Use Case: Deploying a Llama 3-70B model for a customer-facing chatbot requires low latency and high throughput. TensorRT-LLM can compile and serve this model to handle thousands of requests per second with minimal delay.
Quick Start
Use the tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with tensor parallelism across 4 GPUs.
Dependency Matrix
Required Modules
tensorrt-llmtorch
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: tensorrt-llm Download link: https://github.com/Aum08Desai/hermes-research-agent/archive/main.zip#tensorrt-llm Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.