Inference Gateway: Unifying LLM Providers with a High-Performance API

Introduction

The Inference Gateway is an open-source, cloud-native, high-performance proxy server designed to unify access to various language model APIs. It simplifies the configuration and interaction with multiple LLMs, enabling easy use of Mixture of Experts. By providing a single OpenAI-compatible API endpoint, it allows seamless integration with providers ranging from local solutions like Ollama to major cloud platforms such as OpenAI, Groq, Cohere, Anthropic, Cloudflare, and DeepSeek.

Why Use and Key Benefits

Inference Gateway addresses the challenges of managing diverse LLM providers by offering a robust set of features:

Unified API: Interact with over a dozen LLM providers, including OpenAI, Anthropic, Groq, Cohere, Ollama, and more, through a single OpenAI-compatible endpoint. This simplifies development and reduces vendor lock-in.
Tool-use Support and MCP Integration: Leverage function calling capabilities across supported providers and benefit from full Model Context Protocol (MCP) integration. Tools from MCP servers are automatically discovered and exposed to LLMs, and the gateway can even serve as an MCP server itself.
Guardrails: Enhance security and compliance with OPA/Rego policies, secret and PII detection, and optional external guardrail services applied to requests, responses, and MCP tool calls.
Streaming and Multimodal Capabilities: Support real-time token streaming from all providers and process images alongside text with vision-capable models.
Cloud-Native and Lightweight: Designed for modern cloud environments, it offers first-class container support for Docker and Kubernetes, including a dedicated Kubernetes Operator. The gateway is lightweight, with a minimal resource footprint.
Observability: Gain deep insights into performance and usage with comprehensive OpenTelemetry metrics, including Prometheus integration and an OTLP push endpoint.
Privacy First: As a self-hosted solution licensed under Apache 2.0, it ensures zero data collection, prioritizing user privacy and control.
Portability and Flexibility: Avoid vendor lock-in by easily switching between cloud providers or integrating self-hosted LLMs, giving organizations flexibility and data sovereignty.

Installation

For production deployments, running Inference Gateway as a container via Docker or Kubernetes is recommended for better isolation and simplified management.

Using Install Script

The easiest way to install the Inference Gateway is using the automated install script:

curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | bash

To install a specific version:

curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | VERSION=v0.22.3 bash

Manual Download

Download pre-built binaries directly from the releases page.

Download the appropriate archive for your platform.

Extract the binary:

tar -xzf inference-gateway_<OS>_<ARCH>.tar.gz

Move to a directory in your PATH:

sudo mv inference-gateway /usr/local/bin/
chmod +x /usr/local/bin/inference-gateway

Verify Installation

inference-gateway --version

Running the Gateway

Once installed, start the gateway with your configuration:

export OPENAI_API_KEY="your-api-key"
inference-gateway

For detailed configuration options, see the Configurations documentation.

Examples

The Inference Gateway simplifies interaction with various LLM providers. Here are a few examples:

Basic Chat Completion

Send a request to an OpenAI model through the gateway:

curl -X POST http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "openai/gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a pirate."
      },
      {
        "role": "user",
        "content": "Hello, world! How are you doing today?"
      }
    ]
  }'

Image Generation

Generate images using an OpenAI-compatible endpoint (requires IMAGES_ENABLED=true):

curl -X POST http://localhost:8080/v1/images/generations \
  -d '{
    "model": "openai/gpt-image-1",
    "prompt": "A pirate ship sailing into a neon sunset",
    "n": 1,
    "size": "1024x1024"
  }'

Text to Speech

Convert text to speech (requires AUDIO_ENABLED=true):

curl -X POST http://localhost:8080/v1/audio/speech \
  -d '{
    "model": "openai/gpt-4o-mini-tts",
    "input": "Ahoy! Welcome aboard the Inference Gateway.",
    "voice": "alloy"
  }' -o speech.mp3

For more detailed examples, refer to the examples directory in the repository.

Inference Gateway: Unifying LLM Providers with a High-Performance API

Summary

Repository Information

Topics

Use at your own risk