michaelfeil

infinity: Serve Embedding and Reranking Models via API

Infinity is a Python serving engine for text embeddings, reranking, and selected image, audio, and late-interaction models. It provides an OpenAI-aligned REST API with multiple inference backends for teams deploying models from Hugging Face.

View on GitHub Compare
Stars
2.9k
Forks
209
Stars, last 30 days
+25

Our take

Health score

26/100

How it is scored
Activity
19
Community
42
Issues
21
Pull requests
20

+25 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

2,988

Forks

222

Change since Nov 17, 2026: 0 stars·0 forks

Overview

Infinity runs embedding and reranking models behind a REST API, addressing the need to host these inference workloads rather than rely on a hosted provider. It also supports selected multimodal and late-interaction tasks, including CLIP, CLAP, ColBERT-style, and ColPali-style models.

The project is aimed at teams building search, retrieval-augmented generation, or other applications that need model inference as a service. Its API is aligned with OpenAI's embeddings specification, and it can serve multiple models from one process.

Key Features

  • Serves text embedding, reranking, and text classification models.
  • Supports selected image-text, audio-text, ColBERT-style, and ColPali-style embedding tasks.
  • Offers PyTorch, Optimum (ONNX/TensorRT), and CTranslate2 inference engines, with hardware options depending on backend and setup.
  • Uses dynamic batching and worker threads for tokenization.
  • Exposes a FastAPI REST API and an asynchronous Python API.
  • Provides CLI and environment-variable configuration, including launching multiple models.
  • Offers Docker images for standard and specialized deployments.

Use Cases

  • Search and recommendation teams can generate embeddings from self-hosted models for indexing and query processing.
  • RAG developers can deploy embedding and reranking endpoints alongside their application and vector database.
  • Platform teams can offer shared inference endpoints for multiple models through one service.
  • Applications working with image or audio search can serve supported CLIP- or CLAP-based models.
  • Python developers can call model engines directly through the asynchronous API instead of making REST requests.

Getting Started

Install the package and launch a model:

pip install infinity-emb[all]
infinity_emb v2 --model-id BAAI/bge-small-en-v1.5

See the README and documentation for Docker, backend, hardware, and API configuration details.

Considerations

  • Model compatibility depends on the selected inference engine. For example, the README describes CTranslate2 support as limited to BERT models and Optimum use as requiring an ONNX model.
  • Hardware acceleration is not automatic. GPU deployments require compatible drivers, libraries, and container configuration; supported options vary by backend.
  • The repository describes some multimodal and late-interaction support, but lists specific tested model families and also notes unsupported model types.
  • Some specialized Docker images are not built through CI/CD, and the README recommends pinning an exact version for those images.
  • Serving models locally means operators must provision compute, manage model files and caching, and secure the API as appropriate for their deployment.

Found this useful?

Share it with someone who would like infinity.

Comparisons