LitServe: Build Custom AI Inference Servers in Python

Overview

LitServe is a Python framework for turning custom model or AI pipeline logic into an inference server. It provides the serving layer around your code, while letting you decide how models are initialized, how requests are processed, and what responses are returned. It is aimed at teams that need more flexibility than a single-model serving abstraction provides, such as for agents, RAG systems, or pipelines that combine multiple models.

The project supports local and self-managed deployment, and also documents deployment through Lightning AI. It is built on FastAPI and focuses on AI-serving concerns such as batching, streaming, concurrency, and GPU support. LitServe is a framework for building a serving system, not a ready-made LLM runtime like vLLM or Ollama.

Key Features

• Define custom inference behavior in Python through the LitAPI interface.
• Initialize and combine multiple models, databases, or other components in one service.
• Configure batching and streaming for inference requests.
• Support asynchronous request handling and worker scaling.
• Serve varied workloads, including text, vision, audio, classical ML, agents, and RAG pipelines.
• Deploy locally, self-host, or use the documented Lightning AI deployment path.
• Provide OpenAPI and OpenAI compatibility, plus MCP server support.

Use Cases

• ML engineers can expose a PyTorch or other model behind an API while retaining control of preprocessing and response logic.
• Teams building RAG or agent services can combine model calls, retrieval, and custom orchestration in a single Python server.
• Developers serving multiple models or modalities can route and process requests within one custom inference pipeline.
• Operators who need batching, streaming, or GPU-aware serving can use LitServe as a customizable layer, then choose self-hosting or a managed deployment path.

Getting Started

Install the package:

pip install litserve

See the README for a minimal server example, deployment options, and further documentation.

Alternatives

• infinity: Infinity serves embeddings and rerankers through an OpenAI-aligned REST API, while LitServe is a general framework for custom inference APIs.

Considerations

• LitServe provides building blocks and flexibility, so you define the inference and orchestration logic yourself. It is not a turnkey model runtime.
• High-performance LLM serving may require integrating an engine such as vLLM and implementing optimizations such as KV caching.
• GPU use and autoscaling depend on the deployment environment. The README distinguishes self-managed infrastructure from features offered by Lightning AI.
• The repository reports 62 open issues. Review current issues and documentation when evaluating fit for production workloads.