huggingface

text-generation-inference: Serve Large Language Models

A toolkit for serving large language models through text-generation APIs. It provides GPU-oriented inference features such as continuous batching, token streaming, tensor parallelism, and quantization, but is now in maintenance mode.

PythonApache-2.0LLMMachine LearningAPI
View on GitHub Compare
Stars
10.9k
Forks
1.3k
Stars, last 30 days
-1

Our take

Health score

41/100

How it is scored
Activity
31
Community
90
Issues
21
Pull requests
20

-1 stars in the last 30 days

  • More contributors than 78% of the projects we track

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

10,876

Forks

1,288

Change since Nov 15, 2026: -2 stars·0 forks

Overview

Text Generation Inference (TGI) is a server for deploying and serving large language models. It exposes text generation over HTTP and supports common open model architectures, helping teams run inference workloads without assembling their own serving stack.

TGI combines a Rust launcher and routing components with Python model-serving code and gRPC. It is most relevant to teams operating GPU-backed inference services that need features such as streaming responses, batching, or multi-GPU execution. The project is in maintenance mode, so new deployments should weigh its capabilities against the inference engines recommended in its README, including vLLM and SGLang.

Key Features

  • Serves supported causal and sequence-to-sequence language models through HTTP APIs, including a chat-completions-compatible endpoint.
  • Streams generated tokens to clients using Server-Sent Events.
  • Uses continuous batching to combine incoming generation requests.
  • Supports tensor parallelism for inference across multiple GPUs.
  • Offers quantization options, including bitsandbytes, GPTQ, AWQ, Marlin, EETQ, and FP8.
  • Provides Prometheus metrics and OpenTelemetry distributed tracing for operations.
  • Includes controls for generation, such as sampling parameters, stop sequences, logits processing, and constrained JSON output.

Use Cases

  • ML platform teams can expose supported open-weight models as an HTTP generation service for internal applications.
  • Infrastructure engineers can serve GPU-hosted models for chat or text generation while using streaming and batching to handle concurrent requests.
  • Teams with multi-GPU servers can use tensor parallelism to run models that benefit from distributing inference across devices.
  • Developers evaluating quantized model deployments can use its quantization options to reduce model memory requirements, subject to hardware and model compatibility.

Getting Started

The README documents Docker-based setup. For an NVIDIA GPU, install the NVIDIA Container Toolkit, then start a model with the official container:

docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id HuggingFaceH4/zephyr-7b-beta

See the README and documentation for hardware setup, model access, configuration, and API details.

Considerations

  • The project is in maintenance mode. The maintainers say they will accept minor bug fixes, documentation improvements, and lightweight maintenance contributions.
  • GPU inference is the intended use. The README warns that CPU performance may be subpar; NVIDIA and AMD GPU setups have different container requirements.
  • Model architecture and hardware compatibility matter. The project lists optimized architectures, while other architectures are supported on a best-effort basis.
  • Multi-GPU use may require shared memory for NCCL communication. The Docker quick start allocates shared memory, and disabling NCCL shared memory can affect performance.
  • The README recommends considering vLLM, SGLang, llama.cpp, or MLX for future inference deployments.

Found this useful?

Share it with someone who would like text-generation-inference.

Comparisons