text-generation-inference vs torchchat
LLM inference projects for serving and local execution
text-generation-inference serves supported large language models through HTTP APIs, with GPU-oriented features such as streaming, batching, and tensor parallelism. torchchat focuses on running models locally through Python, C++ and mobile workflows, including model export. Both are archived, but text-generation-inference is in maintenance mode while torchchat is no longer under active development.

text-generation-inference: Serve Large Language Models
A toolkit for serving large language models through text-generation APIs. It provides GPU-oriented inference features such as continuous batching, token streaming, tensor parallelism, and quantization, but is now in maintenance mode.

torchchat: Run PyTorch LLMs on Desktop, Server and Mobile
torchchat is a PyTorch-native codebase for running large language models locally with Python, C++ runners, and mobile apps. It supports desktop, server, iOS, and Android workflows, but is archived and no longer under active development.
| text-generation-inference | torchchat | |
|---|---|---|
| Language | Python | Python |
| License | Apache-2.0 | BSD-3-Clause |
| Stars | 10.9k | 3.6k |
| Forks | 1.3k | 247 |
| Last analyzed | Oct 3, 2026 | Oct 26, 2026 |
Key differences
- text-generation-inference is designed for GPU-backed inference services; torchchat targets local execution on desktop, server, and mobile devices.
- text-generation-inference emphasizes API serving, continuous batching, token streaming, and multi-GPU inference; torchchat offers Python and C++ runners, model export, and ExecuTorch workflows.
- text-generation-inference uses an Apache-2.0 license; torchchat uses BSD-3-Clause.
- text-generation-inference supports quantization options including GPTQ, AWQ, and FP8; torchchat includes model export and quantization workflows for local deployment.
- text-generation-inference is in maintenance mode and accepts minor fixes and documentation improvements; torchchat is archived and no longer under active development.
Choose text-generation-inference if you…
- need an HTTP service for supported language models with streaming or continuous batching.
- want to use GPU-oriented features such as tensor parallelism or quantization.
- prefer a project that still accepts minor maintenance contributions.
Choose torchchat if you…
- want to explore local LLM execution through Python or C++ runners.
- are prototyping mobile deployment with ExecuTorch on iOS or Android.
- need to examine model export workflows for desktop, server, or mobile targets.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.