infinity: Serve Embedding and Reranking Models via API

Overview

Infinity runs embedding and reranking models behind a REST API, addressing the need to host these inference workloads rather than rely on a hosted provider. It also supports selected multimodal and late-interaction tasks, including CLIP, CLAP, ColBERT-style, and ColPali-style models.

The project is aimed at teams building search, retrieval-augmented generation, or other applications that need model inference as a service. Its API is aligned with OpenAI's embeddings specification, and it can serve multiple models from one process.

Key Features

• Serves text embedding, reranking, and text classification models.
• Supports selected image-text, audio-text, ColBERT-style, and ColPali-style embedding tasks.
• Offers PyTorch, Optimum (ONNX/TensorRT), and CTranslate2 inference engines, with hardware options depending on backend and setup.
• Uses dynamic batching and worker threads for tokenization.
• Exposes a FastAPI REST API and an asynchronous Python API.
• Provides CLI and environment-variable configuration, including launching multiple models.
• Offers Docker images for standard and specialized deployments.

Use Cases

• Search and recommendation teams can generate embeddings from self-hosted models for indexing and query processing.
• RAG developers can deploy embedding and reranking endpoints alongside their application and vector database.
• Platform teams can offer shared inference endpoints for multiple models through one service.
• Applications working with image or audio search can serve supported CLIP- or CLAP-based models.
• Python developers can call model engines directly through the asynchronous API instead of making REST requests.

Project Facts

• Language: Python
• License: MIT
• Stars: 2.9k
• Forks: 209
• Topics: bert-embeddings, llm, text-embeddings
• Archived: false

Getting Started

Install the package and launch a model:

pip install infinity-emb[all]
infinity_emb v2 --model-id BAAI/bge-small-en-v1.5

See the README and documentation for Docker, backend, hardware, and API configuration details.

Considerations

• Model compatibility depends on the selected inference engine. For example, the README describes CTranslate2 support as limited to BERT models and Optimum use as requiring an ONNX model.
• Hardware acceleration is not automatic. GPU deployments require compatible drivers, libraries, and container configuration; supported options vary by backend.
• The repository describes some multimodal and late-interaction support, but lists specific tested model families and also notes unsupported model types.
• Some specialized Docker images are not built through CI/CD, and the README recommends pinning an exact version for those images.
• Serving models locally means operators must provision compute, manage model files and caching, and secure the API as appropriate for their deployment.