# Inference Gateway: Unifying LLM Providers with a High-Performance API

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/inference-gateway-inference-gateway
Generated for open source discovery and AI-assisted research.

Inference Gateway is an open-source, cloud-native, high-performance proxy server designed to unify access to various language model APIs. It provides a single OpenAI-compatible endpoint, simplifying interactions with multiple LLM providers, from local solutions like Ollama to major cloud platforms. This gateway enables seamless integration and management of diverse AI models, enhancing portability and data privacy.

GitHub: https://github.com/inference-gateway/inference-gateway
OSRepos URL: https://osrepos.com/repo/inference-gateway-inference-gateway

## Summary

Inference Gateway is an open-source, cloud-native, high-performance proxy server designed to unify access to various language model APIs. It provides a single OpenAI-compatible endpoint, simplifying interactions with multiple LLM providers, from local solutions like Ollama to major cloud platforms. This gateway enables seamless integration and management of diverse AI models, enhancing portability and data privacy.

## Topics

- LLM Gateway
- OpenAI API Proxy
- Cloud-Native AI
- Go
- Kubernetes
- Open Source
- Inference API
- Model Context Protocol

## Repository Information

Last analyzed by OSRepos: Fri Sep 25 2026 09:54:36 GMT+0100 (Western European Summer Time)
Detail views: 1
GitHub clicks: 0

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

The Inference Gateway is an open-source, cloud-native, high-performance proxy server designed to unify access to various language model APIs. It simplifies the configuration and interaction with multiple LLMs, enabling easy use of Mixture of Experts. By providing a single OpenAI-compatible API endpoint, it allows seamless integration with providers ranging from local solutions like Ollama to major cloud platforms such as OpenAI, Groq, Cohere, Anthropic, Cloudflare, and DeepSeek.

## Why Use and Key Benefits

Inference Gateway addresses the challenges of managing diverse LLM providers by offering a robust set of features:

*   **Unified API**: Interact with over a dozen LLM providers, including OpenAI, Anthropic, Groq, Cohere, Ollama, and more, through a single OpenAI-compatible endpoint. This simplifies development and reduces vendor lock-in.
*   **Tool-use Support and MCP Integration**: Leverage function calling capabilities across supported providers and benefit from full Model Context Protocol (MCP) integration. Tools from MCP servers are automatically discovered and exposed to LLMs, and the gateway can even serve as an MCP server itself.
*   **Guardrails**: Enhance security and compliance with OPA/Rego policies, secret and PII detection, and optional external guardrail services applied to requests, responses, and MCP tool calls.
*   **Streaming and Multimodal Capabilities**: Support real-time token streaming from all providers and process images alongside text with vision-capable models.
*   **Cloud-Native and Lightweight**: Designed for modern cloud environments, it offers first-class container support for Docker and Kubernetes, including a dedicated Kubernetes Operator. The gateway is lightweight, with a minimal resource footprint.
*   **Observability**: Gain deep insights into performance and usage with comprehensive OpenTelemetry metrics, including Prometheus integration and an OTLP push endpoint.
*   **Privacy First**: As a self-hosted solution licensed under Apache 2.0, it ensures zero data collection, prioritizing user privacy and control.
*   **Portability and Flexibility**: Avoid vendor lock-in by easily switching between cloud providers or integrating self-hosted LLMs, giving organizations flexibility and data sovereignty.

## Installation

For production deployments, running Inference Gateway as a container via Docker or Kubernetes is recommended for better isolation and simplified management.

### Using Install Script

The easiest way to install the Inference Gateway is using the automated install script:

bash
curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | bash


To install a specific version:

bash
curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | VERSION=v0.22.3 bash


### Manual Download

Download pre-built binaries directly from the [releases page](https://github.com/inference-gateway/inference-gateway/releases).

1.  Download the appropriate archive for your platform.
2.  Extract the binary:
    bash
    tar -xzf inference-gateway_<OS>_<ARCH>.tar.gz
    
3.  Move to a directory in your PATH:
    bash
    sudo mv inference-gateway /usr/local/bin/
    chmod +x /usr/local/bin/inference-gateway
    

### Verify Installation

bash
inference-gateway --version


### Running the Gateway

Once installed, start the gateway with your configuration:

bash
export OPENAI_API_KEY="your-api-key"
inference-gateway


For detailed configuration options, see the [Configurations documentation](https://github.com/inference-gateway/inference-gateway/blob/main/Configurations.md).

## Examples

The Inference Gateway simplifies interaction with various LLM providers. Here are a few examples:

### Basic Chat Completion

Send a request to an OpenAI model through the gateway:

bash
curl -X POST http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "openai/gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a pirate."
      },
      {
        "role": "user",
        "content": "Hello, world! How are you doing today?"
      }
    ]
  }'


### Image Generation

Generate images using an OpenAI-compatible endpoint (requires `IMAGES_ENABLED=true`):

bash
curl -X POST http://localhost:8080/v1/images/generations \
  -d '{
    "model": "openai/gpt-image-1",
    "prompt": "A pirate ship sailing into a neon sunset",
    "n": 1,
    "size": "1024x1024"
  }'


### Text to Speech

Convert text to speech (requires `AUDIO_ENABLED=true`):

bash
curl -X POST http://localhost:8080/v1/audio/speech \
  -d '{
    "model": "openai/gpt-4o-mini-tts",
    "input": "Ahoy! Welcome aboard the Inference Gateway.",
    "voice": "alloy"
  }' -o speech.mp3


For more detailed examples, refer to the [examples directory](https://github.com/inference-gateway/inference-gateway/tree/main/examples) in the repository.

## Links

*   **GitHub Repository**: [inference-gateway/inference-gateway](https://github.com/inference-gateway/inference-gateway)
*   **Official Documentation**: [docs.inference-gateway.com](https://docs.inference-gateway.com)
*   **TypeScript SDK**: [inference-gateway/typescript-sdk](https://github.com/inference-gateway/typescript-sdk)
*   **Go SDK**: [inference-gateway/go-sdk](https://github.com/inference-gateway/go-sdk)
*   **Python SDK**: [inference-gateway/python-sdk](https://github.com/inference-gateway/python-sdk)
*   **Rust SDK**: [inference-gateway/rust-sdk](https://github.com/inference-gateway/rust-sdk)
*   **CLI Tool**: [inference-gateway/cli](https://github.com/inference-gateway/cli)