{"name":"Inference Gateway: Unifying LLM Providers with a High-Performance API","description":"Inference Gateway is an open-source, cloud-native, high-performance proxy server designed to unify access to various language model APIs. It provides a single OpenAI-compatible endpoint, simplifying interactions with multiple LLM providers, from local solutions like Ollama to major cloud platforms. This gateway enables seamless integration and management of diverse AI models, enhancing portability and data privacy.","github":"https://github.com/inference-gateway/inference-gateway","url":"https://osrepos.com/repo/inference-gateway-inference-gateway","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/inference-gateway-inference-gateway","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/inference-gateway-inference-gateway.md","json":"https://osrepos.com/repo/inference-gateway-inference-gateway.json","topics":["LLM Gateway","OpenAI API Proxy","Cloud-Native AI","Go","Kubernetes","Open Source","Inference API","Model Context Protocol"],"keywords":["LLM Gateway","OpenAI API Proxy","Cloud-Native AI","Go","Kubernetes","Open Source","Inference API","Model Context Protocol"],"stars":null,"summary":"Inference Gateway is an open-source, cloud-native, high-performance proxy server designed to unify access to various language model APIs. It provides a single OpenAI-compatible endpoint, simplifying interactions with multiple LLM providers, from local solutions like Ollama to major cloud platforms. This gateway enables seamless integration and management of diverse AI models, enhancing portability and data privacy.","content":"## Introduction\n\nThe Inference Gateway is an open-source, cloud-native, high-performance proxy server designed to unify access to various language model APIs. It simplifies the configuration and interaction with multiple LLMs, enabling easy use of Mixture of Experts. By providing a single OpenAI-compatible API endpoint, it allows seamless integration with providers ranging from local solutions like Ollama to major cloud platforms such as OpenAI, Groq, Cohere, Anthropic, Cloudflare, and DeepSeek.\n\n## Why Use and Key Benefits\n\nInference Gateway addresses the challenges of managing diverse LLM providers by offering a robust set of features:\n\n*   **Unified API**: Interact with over a dozen LLM providers, including OpenAI, Anthropic, Groq, Cohere, Ollama, and more, through a single OpenAI-compatible endpoint. This simplifies development and reduces vendor lock-in.\n*   **Tool-use Support and MCP Integration**: Leverage function calling capabilities across supported providers and benefit from full Model Context Protocol (MCP) integration. Tools from MCP servers are automatically discovered and exposed to LLMs, and the gateway can even serve as an MCP server itself.\n*   **Guardrails**: Enhance security and compliance with OPA/Rego policies, secret and PII detection, and optional external guardrail services applied to requests, responses, and MCP tool calls.\n*   **Streaming and Multimodal Capabilities**: Support real-time token streaming from all providers and process images alongside text with vision-capable models.\n*   **Cloud-Native and Lightweight**: Designed for modern cloud environments, it offers first-class container support for Docker and Kubernetes, including a dedicated Kubernetes Operator. The gateway is lightweight, with a minimal resource footprint.\n*   **Observability**: Gain deep insights into performance and usage with comprehensive OpenTelemetry metrics, including Prometheus integration and an OTLP push endpoint.\n*   **Privacy First**: As a self-hosted solution licensed under Apache 2.0, it ensures zero data collection, prioritizing user privacy and control.\n*   **Portability and Flexibility**: Avoid vendor lock-in by easily switching between cloud providers or integrating self-hosted LLMs, giving organizations flexibility and data sovereignty.\n\n## Installation\n\nFor production deployments, running Inference Gateway as a container via Docker or Kubernetes is recommended for better isolation and simplified management.\n\n### Using Install Script\n\nThe easiest way to install the Inference Gateway is using the automated install script:\n\nbash\ncurl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | bash\n\n\nTo install a specific version:\n\nbash\ncurl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | VERSION=v0.22.3 bash\n\n\n### Manual Download\n\nDownload pre-built binaries directly from the [releases page](https://github.com/inference-gateway/inference-gateway/releases).\n\n1.  Download the appropriate archive for your platform.\n2.  Extract the binary:\n    bash\n    tar -xzf inference-gateway_<OS>_<ARCH>.tar.gz\n    \n3.  Move to a directory in your PATH:\n    bash\n    sudo mv inference-gateway /usr/local/bin/\n    chmod +x /usr/local/bin/inference-gateway\n    \n\n### Verify Installation\n\nbash\ninference-gateway --version\n\n\n### Running the Gateway\n\nOnce installed, start the gateway with your configuration:\n\nbash\nexport OPENAI_API_KEY=\"your-api-key\"\ninference-gateway\n\n\nFor detailed configuration options, see the [Configurations documentation](https://github.com/inference-gateway/inference-gateway/blob/main/Configurations.md).\n\n## Examples\n\nThe Inference Gateway simplifies interaction with various LLM providers. Here are a few examples:\n\n### Basic Chat Completion\n\nSend a request to an OpenAI model through the gateway:\n\nbash\ncurl -X POST http://localhost:8080/v1/chat/completions \\\n  -d '{\n    \"model\": \"openai/gpt-3.5-turbo\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a pirate.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello, world! How are you doing today?\"\n      }\n    ]\n  }'\n\n\n### Image Generation\n\nGenerate images using an OpenAI-compatible endpoint (requires `IMAGES_ENABLED=true`):\n\nbash\ncurl -X POST http://localhost:8080/v1/images/generations \\\n  -d '{\n    \"model\": \"openai/gpt-image-1\",\n    \"prompt\": \"A pirate ship sailing into a neon sunset\",\n    \"n\": 1,\n    \"size\": \"1024x1024\"\n  }'\n\n\n### Text to Speech\n\nConvert text to speech (requires `AUDIO_ENABLED=true`):\n\nbash\ncurl -X POST http://localhost:8080/v1/audio/speech \\\n  -d '{\n    \"model\": \"openai/gpt-4o-mini-tts\",\n    \"input\": \"Ahoy! Welcome aboard the Inference Gateway.\",\n    \"voice\": \"alloy\"\n  }' -o speech.mp3\n\n\nFor more detailed examples, refer to the [examples directory](https://github.com/inference-gateway/inference-gateway/tree/main/examples) in the repository.\n\n## Links\n\n*   **GitHub Repository**: [inference-gateway/inference-gateway](https://github.com/inference-gateway/inference-gateway)\n*   **Official Documentation**: [docs.inference-gateway.com](https://docs.inference-gateway.com)\n*   **TypeScript SDK**: [inference-gateway/typescript-sdk](https://github.com/inference-gateway/typescript-sdk)\n*   **Go SDK**: [inference-gateway/go-sdk](https://github.com/inference-gateway/go-sdk)\n*   **Python SDK**: [inference-gateway/python-sdk](https://github.com/inference-gateway/python-sdk)\n*   **Rust SDK**: [inference-gateway/rust-sdk](https://github.com/inference-gateway/rust-sdk)\n*   **CLI Tool**: [inference-gateway/cli](https://github.com/inference-gateway/cli)","metrics":{"detailViews":1,"githubClicks":0},"dates":{"published":null,"modified":"2026-09-25T08:54:36.000Z"}}