text-generation-inference vs TensorRT-LLM
LLM inference projects compared
text-generation-inference and TensorRT-LLM both support running language models for inference on GPUs. text-generation-inference provides an HTTP serving toolkit with features such as streaming and continuous batching, while TensorRT-LLM focuses on optimizing and customizing inference on NVIDIA GPUs.

text-generation-inference: Serve Large Language Models
A toolkit for serving large language models through text-generation APIs. It provides GPU-oriented inference features such as continuous batching, token streaming, tensor parallelism, and quantization, but is now in maintenance mode.

TensorRT-LLM: Optimize LLM Inference on NVIDIA GPUs
NVIDIA TensorRT-LLM is a Python-native framework for optimizing inference for large language and visual generation models on NVIDIA GPUs. It combines specialized kernels and runtime optimizations with support for single-GPU through multi-node deployments.
| text-generation-inference | TensorRT-LLM | |
|---|---|---|
| Language | Python | Python |
| License | Apache-2.0 | — |
| Stars | 10.9k | 15k |
| Forks | 1.3k | 2.9k |
| Last analyzed | Oct 3, 2026 | Oct 29, 2026 |
Key differences
- text-generation-inference serves supported models through HTTP APIs; TensorRT-LLM provides a Python framework and runtime for LLM and visual-generation inference.
- text-generation-inference includes token streaming and continuous batching; TensorRT-LLM emphasizes specialized kernels and runtime optimizations such as speculative decoding and prefill-decode disaggregation.
- text-generation-inference supports NVIDIA and AMD GPU setups, with hardware-specific requirements; TensorRT-LLM targets NVIDIA GPUs.
- text-generation-inference is in maintenance mode and archived; TensorRT-LLM is not archived.
- text-generation-inference uses Apache-2.0; TensorRT-LLM's repository metadata reports NOASSERTION.
- Both list Python as a language, while TensorRT-LLM also describes PyTorch-native components that developers can modify and extend.
Choose text-generation-inference if you…
- need HTTP text-generation APIs with streaming or continuous batching.
- want options such as tensor parallelism and quantization for supported models.
- are maintaining or evaluating an existing deployment and can account for the project's maintenance mode.
Choose TensorRT-LLM if you…
- deploy on NVIDIA GPUs and want to tune inference kernels or runtime behavior.
- need single-GPU, multi-GPU, or multi-node inference configurations.
- want Python and PyTorch-native components to adapt supported models or runtime behavior.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.