# llm-d Router: Intelligent Orchestration for Multi-Phase LLM Inference

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/llm-d-llm-d-router
Generated for open source discovery and AI-assisted research.

The `llm-d-router` is a sophisticated Go service designed as an intelligent entry point for large language model (LLM) inference requests. It orchestrates complex, multi-phase LLM inference pipelines across specialized worker pools, routing requests through an Inference Gateway to disaggregated vLLM workers. By exposing OpenAI-compatible APIs, it simplifies integration and leverages Kubernetes for scalable, efficient deployments.

GitHub: https://github.com/llm-d/llm-d-router
OSRepos URL: https://osrepos.com/repo/llm-d-llm-d-router

## Summary

The `llm-d-router` is a sophisticated Go service designed as an intelligent entry point for large language model (LLM) inference requests. It orchestrates complex, multi-phase LLM inference pipelines across specialized worker pools, routing requests through an Inference Gateway to disaggregated vLLM workers. By exposing OpenAI-compatible APIs, it simplifies integration and leverages Kubernetes for scalable, efficient deployments.

## Topics

- ai
- gateway-api
- inference
- kubernetes
- networking
- Go
- LLM
- Orchestration

## Repository Information

Last analyzed by OSRepos: Fri Sep 25 2026 12:53:21 GMT+0100 (Western European Summer Time)
Detail views: 0
GitHub clicks: 0

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

The `llm-d-router` is a sophisticated Go service that serves as an intelligent entry point for large language model (LLM) inference requests. It specializes in orchestrating multi-phase LLM inference pipelines, such as Encode, Prefill, and Decode, across specialized worker pools. By exposing OpenAI-compatible APIs, it simplifies integration for developers, routing requests through an Inference Gateway to disaggregated vLLM workers. This project is built to leverage Kubernetes for scalable and efficient deployment of LLM inference infrastructure.

## Why Use and Key Benefits

`llm-d-router` offers several compelling advantages for managing LLM inference:

*   **Intelligent Orchestration:** It efficiently manages complex LLM inference workflows, coordinating different stages and specialized worker types to optimize performance and resource utilization.
*   **OpenAI API Compatibility:** Provides familiar API endpoints, including `/v1/chat/completions` and `/v1/completions`, ensuring seamless integration with existing LLM applications and tools.
*   **Scalability and Disaggregation:** Designed for Kubernetes environments, it facilitates highly scalable and efficient inference by routing requests to disaggregated vLLM workers.
*   **Flexible Configuration:** Offers extensive configuration options via YAML files, allowing for deep customization of pipeline behavior, environment overrides, and connector selection.
*   **Robust Testing Framework:** Includes comprehensive unit and end-to-end tests, supporting various Endpoint Picker (EPP) topologies to ensure reliability and performance in diverse deployment scenarios.

## Installation

To get started with `llm-d-router`, follow these quick steps:

bash
# Build the coordinator binary
make -f Makefile.coord.mk build

# Run with the default configuration
make -f Makefile.coord.mk run

# Run with a custom configuration file
./bin/coordinator --config path/to/config.yaml


The listener serves HTTPS on port `8080` by default. Refer to the official documentation for TLS configuration.

## Examples

### API Endpoints

The `llm-d-router` exposes several API endpoints, compatible with OpenAI's API:

*   `POST /v1/chat/completions`: OpenAI Chat Completions API
*   `POST /v1/completions`: OpenAI Completions API
*   `GET /healthz`: Health check endpoint
*   `GET /readyz`: Readiness check endpoint

Both completion endpoints support `"stream": true` for Server-Sent Events streaming.

### Docker Usage

You can easily build and run `llm-d-router` using Docker:

bash
# Build the Docker image
docker build -t coordinator -f Dockerfile.coordinator .

# Run the Docker container, mapping port 8080 and mounting configuration
docker run -p 8080:8080 -v $(pwd)/config/coordinator:/config/coordinator coordinator


## Links

*   [GitHub Repository](https://github.com/llm-d/llm-d-router)
*   [Coordinator Architecture Documentation](https://github.com/llm-d/llm-d-router/blob/main/docs/coordinator_architecture.md)
*   [Communication Wire Formats Documentation](https://github.com/llm-d/llm-d-router/blob/main/docs/communication.md)