NVIDIA

TensorRT-LLM: Optimize LLM Inference on NVIDIA GPUs

NVIDIA TensorRT-LLM is a Python-native framework for optimizing inference for large language and visual generation models on NVIDIA GPUs. It combines specialized kernels and runtime optimizations with support for single-GPU through multi-node deployments.

ActivePythonLLMAIGenerative AI
Stars
15k
Forks
2.9k
Stars, last 30 days
+240
Last commit
today
Contributors
100+

Our take

Activeregular commits and releases

Health score

62/100

How it is scored
Activity
85
Community
94
Issues
21
Pull requests
49

+240 stars in the last 30 days

  • Releases more often than 87% of the projects we track
  • More contributors than 78% of the projects we track
  • Merges more pull requests than 78% of the projects we track

TensorRT-LLM is an actively developed, NVIDIA-GPU-focused inference framework with substantial recent release activity, but adopting it means committing to its specialized hardware and runtime stack.

Good fit if

  • Your inference workloads run on NVIDIA GPUs and you need the framework's kernel and runtime optimizations.
  • You want a Python and PyTorch-based route to customize inference, from one GPU to multi-node deployments.
  • Your serving stack can benefit from integration with NVIDIA Dynamo or Triton Inference Server.

Look elsewhere if

  • You need a framework aimed at GPUs other than NVIDIA's, or a hardware-neutral deployment path.
  • Your project does not use Python or cannot accommodate a GPU-dependent inference stack.
  • You require an unambiguous license determination before adoption, since GitHub reports NOASSERTION while the README identifies Apache 2.0.
All health signals
Last commit2026-10-29 (today)
Commits, last 90 days100+
Releases, last 12 months49 (latest v1.4.0rc2, 2026-10-28, pre-release)
Contributors100+ (top contributor: 6% of commits)
Issues closed, last 90 days0+
Pull requests merged, last 90 days60 (typically merged in 2 days)
Project age3 years

Checked on 2026-10-29 with the GitHub API.

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

15,091

Forks

2,940

Change since Nov 17, 2026: +7 stars·+2 forks

Overview

NVIDIA TensorRT-LLM is a Python-native framework built around NVIDIA TensorRT and PyTorch for optimizing inference for large language models and visual generation models on NVIDIA GPUs. It addresses the performance and deployment demands of inference workloads with specialized kernels, an efficient runtime, and a framework developers can customize.

It supports setups from one GPU to multi-GPU and multi-node deployments, with parallelism strategies and integrations with NVIDIA Dynamo and Triton Inference Server. Its PyTorch-based architecture also lets teams adapt model implementations and runtime behavior rather than treating the serving stack as a fixed black box.

Key Features

  • Specialized kernels for common inference operations, including attention, GEMMs, and mixture-of-experts workloads.
  • Runtime techniques including prefill-decode disaggregation, expert parallelism, and speculative decoding.
  • A high-level Python LLM API for configuring and running inference.
  • Support for single-GPU, multi-GPU, and multi-node inference setups.
  • PyTorch-native model implementations that developers can customize.
  • Integration with NVIDIA Dynamo and Triton Inference Server.
  • Support for inference workloads involving both LLMs and visual generation models.

Use Cases

  • Inference teams tuning latency or throughput for LLM services on NVIDIA GPUs.
  • Platform engineers deploying models across multiple GPUs or nodes, including with Triton or NVIDIA Dynamo.
  • Developers adapting model implementations or runtime behavior to a specific workload using PyTorch code.
  • Teams exploring optimized inference for visual generation models alongside language models.

What you need

Detected in the repository

  • Python (from pyproject.toml)
  • A test suite and automated checks on GitHub Actions

Getting Started

Start with the Quick Start Guide, then consult the installation guide and support matrix for setup and compatibility details. See the README for project documentation links.

License in Plain Words

No standard license was detected. Without a license, the default copyright rules apply: you may read the code, but not reuse it. Check the repository or ask the authors.

Alternatives

  • LocalAI: LocalAI provides a self-hosted, OpenAI-compatible model server across varied hardware, while TensorRT-LLM focuses on optimized inference on NVIDIA GPUs.
  • llama-cpp-python: llama-cpp-python runs GGUF models through llama.cpp and supports multiple hardware backends, while TensorRT-LLM targets optimized NVIDIA GPU inference.
ProjectLanguageLicenseStarsStatus
TensorRT-LLMPython–15kActive
LocalAIGoMIT49.4kActive
llama-cpp-pythonPythonMIT10.6kActive

Considerations

  • This is a specialized NVIDIA GPU inference stack, not a general-purpose runtime for arbitrary hardware. Check the support matrix against your GPU, model, and software environment before planning a deployment.
  • The project is actively maintained, with regular commits and releases and contributions from many developers; the available health data does not provide a median issue-closure time.
  • License information is inconsistent in the supplied repository data: GitHub reports NOASSERTION, while the README displays Apache 2.0. Verify the applicable license in the repository before distributing or incorporating it.

Found this useful?

Share it with someone who would like TensorRT-LLM.

Comparisons