inference-gateway vs llm-d-router
LLM inference gateways compared
inference-gateway provides a common API for routing requests to hosted and local model providers, with optional tools, access controls, and telemetry. llm-d-router selects serving endpoints using signals such as cache locality, load, and priority, for inference workloads running on Kubernetes. The main distinction is provider access and API compatibility versus inference-aware routing within a serving environment.
inference-gateway: Unify LLM Providers Behind One API
A Go-based, cloud-native gateway that puts multiple LLM providers behind one OpenAI-compatible API. It supports cloud and self-hosted models, with integrations for MCP, A2A, guardrails and OpenTelemetry.
llm-d-router: Route LLM Inference with Envoy and Gateway API
llm-d-router is a Go-based router for LLM inference traffic, using an Endpoint Picker with Envoy and the Kubernetes Gateway API. It routes requests using signals such as model-server load and KV-cache locality, with options for standalone or gateway deployments.
| inference-gateway | llm-d-router | |
|---|---|---|
| Language | Go | Go |
| License | Apache-2.0 | Apache-2.0 |
| Stars | 215 | 422 |
| Forks | 48 | 472 |
| Last analyzed | Oct 23, 2026 | Oct 20, 2026 |
Key differences
- inference-gateway offers compatible chat APIs across cloud and local providers; llm-d-router focuses on choosing among inference-serving endpoints based on serving state.
- inference-gateway supports MCP tool integration and optional authentication, guardrails, and telemetry; llm-d-router provides request priorities, performance objectives, and model-name rewrites.
- inference-gateway can be deployed in containers or Kubernetes and routes to separately configured model services; llm-d-router requires an inference-serving setup and a supported proxy integration, with standalone Envoy and Gateway API modes.
- Both projects are written in Go and licensed under Apache-2.0.
- The supplied repository figures report 214 stars and 50 forks for inference-gateway, compared with 373 stars and 410 forks for llm-d-router.
Choose inference-gateway if you…
- need a common API for switching or combining hosted and local model providers.
- want to connect MCP tools centrally or configure gateway-level authentication and observability.
- need a self-hostable routing layer for applications using Docker or Kubernetes.
Choose llm-d-router if you…
- operate LLM serving on Kubernetes and want routing informed by cache locality, load, or request priority.
- use Envoy or Gateway API infrastructure to connect inference backends.
- need request objectives, model-name rewrites for canary or A/B traffic, or coordination for multi-stage inference.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.