TensorRT-LLM: Optimize LLM Inference on NVIDIA GPUs

Overview

TensorRT-LLM is a framework for running large language models and visual-generation models efficiently on NVIDIA GPUs. It combines optimized inference operations with a runtime for orchestrating execution, addressing the performance and deployment needs of GPU-based inference workloads.

It is aimed at developers and infrastructure teams who want to tune or customize inference rather than use a hardware-agnostic serving layer. Its Python-native design supports model and runtime customization, while its APIs cover deployments from one GPU to multiple GPUs or nodes.

Key Features

• Specialized kernels for common inference operations, including attention, GEMMs, and mixture-of-experts workloads.
• Runtime optimizations including speculative decoding and prefill-decode disaggregation.
• Python API for configuring and running supported models.
• Support for single-GPU, multi-GPU, and multi-node inference setups.
• Built-in parallelism strategies for distributing inference work.
• PyTorch-native components that developers can modify and extend.
• Integration with NVIDIA Dynamo and Triton Inference Server.

Use Cases

• Inference platform teams serving LLMs on NVIDIA GPUs that need to tune throughput or latency for their workloads.
• Developers adapting supported model implementations or runtime behavior using Python and PyTorch.
• Operators deploying large models across multiple GPUs or nodes when a single GPU is insufficient.
• Teams evaluating optimizations such as speculative decoding or prefill-decode disaggregation for their serving setup.
• Visual-generation workloads on NVIDIA GPUs that can use the project's supported inference components.

Project Facts

• Language: Python
• License: NOASSERTION
• Stars: 14.8k
• Forks: 2.8k
• Topics: blackwell, cuda, llm-serving, moe, pytorch
• Archived: No

Getting Started

Start with the Quick Start Guide, then check the Installation Guide and supported hardware and models. See the repository README for project details and additional links.

Alternatives

• Text Generation Inference: Text Generation Inference is a general-purpose LLM serving toolkit, while TensorRT-LLM focuses on NVIDIA GPU-optimized inference and distributed execution.

Considerations

• The project targets NVIDIA GPUs, so it is not a general-purpose GPU-agnostic inference framework.
• Multi-GPU and multi-node use introduces deployment and configuration complexity; check the support matrix for hardware and model compatibility.
• The repository metadata reports the license as NOASSERTION. Review the repository's license information before adopting it.
• The README says anonymous telemetry is collected by default and documents ways to opt out, including environment variables, a configuration file, a Python option, and CLI flags.