{"name":"llm-d Router: Intelligent Orchestration for Multi-Phase LLM Inference","description":"The `llm-d-router` is a sophisticated Go service designed as an intelligent entry point for large language model (LLM) inference requests. It orchestrates complex, multi-phase LLM inference pipelines across specialized worker pools, routing requests through an Inference Gateway to disaggregated vLLM workers. By exposing OpenAI-compatible APIs, it simplifies integration and leverages Kubernetes for scalable, efficient deployments.","github":"https://github.com/llm-d/llm-d-router","url":"https://osrepos.com/repo/llm-d-llm-d-router","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/llm-d-llm-d-router","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/llm-d-llm-d-router.md","json":"https://osrepos.com/repo/llm-d-llm-d-router.json","topics":["ai","gateway-api","inference","kubernetes","networking","Go","LLM","Orchestration"],"keywords":["ai","gateway-api","inference","kubernetes","networking","Go","LLM","Orchestration"],"stars":null,"summary":"The `llm-d-router` is a sophisticated Go service designed as an intelligent entry point for large language model (LLM) inference requests. It orchestrates complex, multi-phase LLM inference pipelines across specialized worker pools, routing requests through an Inference Gateway to disaggregated vLLM workers. By exposing OpenAI-compatible APIs, it simplifies integration and leverages Kubernetes for scalable, efficient deployments.","content":"## Introduction\n\nThe `llm-d-router` is a sophisticated Go service that serves as an intelligent entry point for large language model (LLM) inference requests. It specializes in orchestrating multi-phase LLM inference pipelines, such as Encode, Prefill, and Decode, across specialized worker pools. By exposing OpenAI-compatible APIs, it simplifies integration for developers, routing requests through an Inference Gateway to disaggregated vLLM workers. This project is built to leverage Kubernetes for scalable and efficient deployment of LLM inference infrastructure.\n\n## Why Use and Key Benefits\n\n`llm-d-router` offers several compelling advantages for managing LLM inference:\n\n*   **Intelligent Orchestration:** It efficiently manages complex LLM inference workflows, coordinating different stages and specialized worker types to optimize performance and resource utilization.\n*   **OpenAI API Compatibility:** Provides familiar API endpoints, including `/v1/chat/completions` and `/v1/completions`, ensuring seamless integration with existing LLM applications and tools.\n*   **Scalability and Disaggregation:** Designed for Kubernetes environments, it facilitates highly scalable and efficient inference by routing requests to disaggregated vLLM workers.\n*   **Flexible Configuration:** Offers extensive configuration options via YAML files, allowing for deep customization of pipeline behavior, environment overrides, and connector selection.\n*   **Robust Testing Framework:** Includes comprehensive unit and end-to-end tests, supporting various Endpoint Picker (EPP) topologies to ensure reliability and performance in diverse deployment scenarios.\n\n## Installation\n\nTo get started with `llm-d-router`, follow these quick steps:\n\nbash\n# Build the coordinator binary\nmake -f Makefile.coord.mk build\n\n# Run with the default configuration\nmake -f Makefile.coord.mk run\n\n# Run with a custom configuration file\n./bin/coordinator --config path/to/config.yaml\n\n\nThe listener serves HTTPS on port `8080` by default. Refer to the official documentation for TLS configuration.\n\n## Examples\n\n### API Endpoints\n\nThe `llm-d-router` exposes several API endpoints, compatible with OpenAI's API:\n\n*   `POST /v1/chat/completions`: OpenAI Chat Completions API\n*   `POST /v1/completions`: OpenAI Completions API\n*   `GET /healthz`: Health check endpoint\n*   `GET /readyz`: Readiness check endpoint\n\nBoth completion endpoints support `\"stream\": true` for Server-Sent Events streaming.\n\n### Docker Usage\n\nYou can easily build and run `llm-d-router` using Docker:\n\nbash\n# Build the Docker image\ndocker build -t coordinator -f Dockerfile.coordinator .\n\n# Run the Docker container, mapping port 8080 and mounting configuration\ndocker run -p 8080:8080 -v $(pwd)/config/coordinator:/config/coordinator coordinator\n\n\n## Links\n\n*   [GitHub Repository](https://github.com/llm-d/llm-d-router)\n*   [Coordinator Architecture Documentation](https://github.com/llm-d/llm-d-router/blob/main/docs/coordinator_architecture.md)\n*   [Communication Wire Formats Documentation](https://github.com/llm-d/llm-d-router/blob/main/docs/communication.md)","metrics":{"detailViews":0,"githubClicks":0},"dates":{"published":null,"modified":"2026-09-25T11:53:21.000Z"}}