inference-gateway: Unify LLM Providers Behind One API

Overview

Inference Gateway is a Go-based proxy that gives clients one OpenAI-compatible API for cloud and self-hosted language models. It routes requests to providers such as OpenAI, Anthropic, Ollama and DeepSeek, reducing the need to change application code when switching providers or combining them.

It is designed for deployments ranging from Docker containers to Kubernetes, including use with the project's Kubernetes operator. The gateway also extends beyond model requests with optional MCP tool integration, A2A delegation, authentication, guardrails and telemetry.

Key Features

• Routes chat-completion requests through a unified API, with provider selection based on the model name or an explicit provider parameter.
• Supports streaming, tool use and vision-capable models across supported providers.
• Connects to MCP servers to discover and execute tools, and can expose the gateway as an MCP server.
• Provides an A2A server that relays calls to configured agents.
• Offers OIDC authentication, authorization and OPA/Rego guardrails.
• Exports OpenTelemetry metrics in Prometheus format and accepts optional OTLP metric pushes.
• Runs as a container or standalone binary and supports Kubernetes deployment.

Use Cases

• Application teams can keep one API integration while trying different hosted LLM providers or moving workloads between them.
• Self-hosting teams can expose models served by Ollama or llama.cpp through the same interface used for cloud models.
• Platform teams can put authentication, policy checks and telemetry in front of internal LLM clients.
• Agent developers can connect MCP tool servers or route A2A calls through a shared gateway rather than configure every client separately.

Getting Started

For a quick binary installation on Linux or macOS:

curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | bash

Set the environment variables for the provider you want to use, then start the gateway with inference-gateway. See the README and documentation for container, Kubernetes and configuration details.

Alternatives

• kong: Kong is a Lua-based general API, LLM, and MCP gateway, while inference-gateway is a Go gateway focused on a unified OpenAI-compatible LLM API.
• helicone: Helicone combines provider routing with LLM observability, including traces and cost tracking; inference-gateway focuses on gateway integrations and routing.
• a3m-router: A3M Router uses task-based Jev System One routing with optional parallel ensembles, unlike inference-gateway’s broader provider and protocol integrations.
• otari: Otari emphasizes virtual keys, budgets, and routing policies for self-hosted teams; inference-gateway highlights MCP, A2A, guardrails, and OpenTelemetry.

Considerations

• The gateway is a routing and integration layer, not a model server. You still need access to the providers and any required credentials or local model services.
• It is actively developed, with frequent commits and releases. The project has contributors beyond its largest contributor, so its work is not classified as concentrated in one person.
• The documented Go runtime is version 1.26.8. Docker is supported; the README recommends containers for production and provides Kubernetes deployment examples.
• Some capabilities are opt-in. For example, vision handling is disabled by default, and MCP exposure, image, audio and video endpoints require configuration.
• Check the documented protocol requirements before adopting MCP integrations. The README specifies a stateless MCP protocol revision and notes that legacy clients or servers may not work.