llm-d Router: Intelligent Orchestration for Multi-Phase LLM Inference

Introduction

The llm-d-router is a sophisticated Go service that serves as an intelligent entry point for large language model (LLM) inference requests. It specializes in orchestrating multi-phase LLM inference pipelines, such as Encode, Prefill, and Decode, across specialized worker pools. By exposing OpenAI-compatible APIs, it simplifies integration for developers, routing requests through an Inference Gateway to disaggregated vLLM workers. This project is built to leverage Kubernetes for scalable and efficient deployment of LLM inference infrastructure.

Why Use and Key Benefits

llm-d-router offers several compelling advantages for managing LLM inference:

• Intelligent Orchestration: It efficiently manages complex LLM inference workflows, coordinating different stages and specialized worker types to optimize performance and resource utilization.
• OpenAI API Compatibility: Provides familiar API endpoints, including /v1/chat/completions and /v1/completions, ensuring seamless integration with existing LLM applications and tools.
• Scalability and Disaggregation: Designed for Kubernetes environments, it facilitates highly scalable and efficient inference by routing requests to disaggregated vLLM workers.
• Flexible Configuration: Offers extensive configuration options via YAML files, allowing for deep customization of pipeline behavior, environment overrides, and connector selection.
• Robust Testing Framework: Includes comprehensive unit and end-to-end tests, supporting various Endpoint Picker (EPP) topologies to ensure reliability and performance in diverse deployment scenarios.

Installation

To get started with llm-d-router, follow these quick steps:

# Build the coordinator binary
make -f Makefile.coord.mk build

# Run with the default configuration
make -f Makefile.coord.mk run

# Run with a custom configuration file
./bin/coordinator --config path/to/config.yaml

The listener serves HTTPS on port 8080 by default. Refer to the official documentation for TLS configuration.

Examples

API Endpoints

The llm-d-router exposes several API endpoints, compatible with OpenAI's API:

• POST /v1/chat/completions: OpenAI Chat Completions API
• POST /v1/completions: OpenAI Completions API
• GET /healthz: Health check endpoint
• GET /readyz: Readiness check endpoint

Both completion endpoints support "stream": true for Server-Sent Events streaming.

Docker Usage

You can easily build and run llm-d-router using Docker:

# Build the Docker image
docker build -t coordinator -f Dockerfile.coordinator .

# Run the Docker container, mapping port 8080 and mounting configuration
docker run -p 8080:8080 -v $(pwd)/config/coordinator:/config/coordinator coordinator

Links

• GitHub Repository
• Coordinator Architecture Documentation
• Communication Wire Formats Documentation