llm-d-router: Route LLM Inference with Envoy and Gateway API

Overview

llm-d-router is a Go-based intelligent entry point for LLM inference traffic. Its Endpoint Picker (EPP) works with proxies such as Envoy through the ext-proc protocol, using signals including server load and KV-cache locality to choose where requests go. It supports standalone deployments with a self-managed Envoy proxy and Gateway API deployments that use an InferencePool behind an HTTPRoute.

The project is aimed at teams operating model-serving infrastructure that need routing decisions to account for inference-specific conditions, rather than relying only on general-purpose load balancing. The EPP also provides request prioritization and flow control, and the repository includes a disaggregation sidecar for coordinating multi-stage inference workflows.

Key Features

• Routes inference requests using Endpoint Picker scoring informed by InferencePool state, including KV-cache locality and current load.
• Integrates with L7 proxies such as Envoy using the ext-proc protocol.
• Supports standalone operation with Envoy as a sidecar or as a separately scalable service.
• Supports Gateway API deployments using an InferencePool, HTTPRoute, and shared Gateway.
• Provides InferenceObjective configuration for request priorities and performance targets.
• Provides InferenceModelRewrite for model-name rewriting, including traffic management for canary rollouts and A/B testing.
• Includes a disaggregation sidecar for coordinating prefill, decode, and encode stages and associated data transfers.

Use Cases

• Model-serving platform teams can route requests according to backend load and KV-cache locality.
• Kubernetes operators can connect inference backends to shared Gateway API infrastructure alongside other workloads.
• Teams evaluating inference-aware routing can run a standalone setup with a self-managed Envoy proxy.
• Serving teams using prefill/decode or encode/prefill/decode stages can use the sidecar to coordinate workers and transfers.
• Teams running canary rollouts or A/B tests can rewrite model names through the request-management API.

Getting Started

The README describes the standalone and Gateway API deployment modes. Start there, then use the Helm chart documentation for deployment configuration. For development, the repository uses Go 1.27.2, as declared in go.mod.

Alternatives

• a3m-router: A3M routes OpenAI-compatible requests across LLM providers by complexity and cost, rather than routing inference traffic among model-server endpoints using load and KV-cache signals.
• otari: Otari routes API requests to LLM providers and adds keys, budgets, and usage tracking, rather than selecting Kubernetes model-server endpoints based on serving signals.

Considerations

• The router is infrastructure software, not a model runtime. Adopting it means integrating it with an inference pool and a compatible proxy or Gateway API setup.
• For self-managed Envoy configurations, the README identifies FULL_DUPLEX_STREAMED as the only supported ext-proc request and response body mode.
• The project is active, with regular commits and releases, and has activity from at least 100 contributors. Its issue and pull-request queues are being handled regularly, though the repository's open issue backlog is substantial.
• The repository is licensed under Apache-2.0. Review that license and your organization's obligations before incorporating or redistributing the code.