llm-d-router: Route Inference Requests Intelligently

Overview

llm-d Router is an intelligent entry point for LLM inference traffic. Its Endpoint Picker (EPP) evaluates request and serving state, including cache locality, load, and priority, to choose where requests should go. This helps operators make routing decisions that account for inference-specific conditions rather than relying only on generic load balancing.

It is aimed at teams operating model-serving workloads in Kubernetes. It can run with a self-managed Envoy proxy or integrate with a Gateway API deployment, and includes APIs for request priorities, performance objectives, and model-name rewrites.

Key Features

• Routes requests using Endpoint Picker scoring informed by InferencePool state.
• Integrates with L7 proxies such as Envoy through the ext-proc protocol.
• Considers KV-cache locality, current load, and request priority when selecting endpoints.
• Supports Standalone Mode with Envoy deployed alongside the EPP or as a separate service.
• Supports Gateway Mode, connecting an InferencePool to HTTPRoute and shared Gateway infrastructure.
• Provides InferenceObjective resources for scheduling goals, priorities, and performance targets.
• Provides InferenceModelRewrite resources for model-name rewriting, including canary and A/B traffic management.
• Includes a disaggregation sidecar for coordinating multi-stage inference workflows such as prefill and decode.

Use Cases

• Kubernetes platform teams can route LLM requests based on live serving conditions rather than distributing traffic without regard to cache locality or load.
• Operators using Gateway API can connect inference backends to shared gateway infrastructure and manage inference alongside other traffic.
• Model-serving teams can prioritize selected requests or express performance goals through InferenceObjective resources.
• Teams rolling out model versions can rewrite model names to direct traffic for canary deployments or A/B testing.
• Serving teams using prefill/decode or encode/prefill/decode stages can coordinate worker communication and cache or embedding transfers with the disaggregation sidecar.

Getting Started

The repository README describes deployment modes and links to the Helm chart documentation. Start there to choose between Standalone Mode and Gateway Mode: llm-d Router README.

Alternatives

• inference-gateway: Routes across cloud and local LLM providers through compatible APIs, rather than using serving signals such as KV-cache locality to route Kubernetes inference traffic.
• kong: Provides a general API gateway with LLM and MCP features, rather than specializing in inference routing based on model-serving signals.
• a3m-router: Routes OpenAI-compatible requests among model providers using heuristic or ML policies, failover, and caching, rather than Kubernetes serving signals.
• otari: Routes requests among LLM providers with policies, virtual keys, and budget controls, rather than directing traffic by serving-pod load or KV-cache locality.

Considerations

• This is a Kubernetes- and proxy-oriented serving component, not a standalone model runner. Deployment requires an inference-serving setup and a supported proxy integration.
• For manually installed or configured Envoy, the README identifies FULL_DUPLEX_STREAMED as the supported request and response body mode for ext-proc.
• Gateway Mode relies on Gateway API and InferencePool resources. Standalone Mode is available when Gateway API infrastructure is not being used.
• The repository is actively maintained and Apache-2.0 licensed; its reported issue count is 358, so review open issues and project documentation when evaluating operational fit.