tensorrt-llm_skill

Overview

This skill optimizes large language model inference using NVIDIA TensorRT to deliver maximum throughput and minimal latency on NVIDIA GPUs. It targets production deployments on A100/H100/GB200-class hardware and supports quantization, multi-GPU scaling, and in-flight batching for real-time and high-throughput use cases.

How this skill works

The skill compiles and runs LLMs with TensorRT kernels, leveraging optimized attention, paged KV cache, CUDA graphs, and hardware-aware quantization (FP8/INT4/FP4) to reduce compute and memory overhead. It exposes APIs for model loading, sampling, and serving, and supports tensor/pipeline parallelism, speculative decoding, and automatic in-flight batching to maximize utilization across GPUs.

When to use it

• Deploying LLMs on NVIDIA GPUs (A100, H100, GB200) for production inference
• Needing 10–100× faster inference than standard PyTorch runtimes
• Serving quantized models (FP8, INT4, FP4) to reduce memory and cost
• Scaling inference across multiple GPUs or nodes with tensor/pipeline parallelism
• Low-latency or real-time generation where per-token latency matters

Best practices

• Use Docker images or pinned pip releases matched to your CUDA and TensorRT versions to avoid compatibility issues
• Precompile and warm caches for production models to eliminate first-request stalls
• Choose quantization (FP8/INT4) after validating accuracy trade-offs on representative inputs
• Tune tensor/pipeline parallelism to match GPU memory and interconnect topology
• Enable in-flight batching and set max_batch_size based on typical traffic to maximize throughput

Example use cases

• Real-time chat assistants requiring sub-20ms token latency on H100
• High-volume API serving with 10k+ tokens/sec throughput using in-flight batching
• Cost-efficient hosting of large models using FP8 quantization to halve memory footprint
• Multi-GPU deployment of Llama-3-70B across 8 A100s for high-concurrency inference
• Edge server farms using trtllm-serve with automatic model download and compilation