Otari: Self-Hosted OpenAI-Compatible LLM Gateway for 40+ Providers

Summary

Otari, from Mozilla AI, is an open-source, self-hosted LLM gateway. It provides a single OpenAI-compatible endpoint to connect with over 40 model providers, offering features like virtual keys, budget enforcement, and usage tracking. This solution empowers users to manage their AI stack with greater control and flexibility.

Use at your own risk

OSRepos shares public repositories for knowledge and discovery only. Any installation, execution, configuration, or use of code from these repositories is the user's own responsibility. Always review the repository, source code, dependencies, licenses, and security implications before running or installing anything. OSRepos is not responsible for issues, damages, or losses resulting from third-party repositories.

Introduction

Otari, from Mozilla AI, is an open-source, self-hosted LLM gateway designed to simplify interaction with various large language model providers. It offers a single OpenAI-compatible endpoint, allowing seamless integration with over 40 different LLM providers. Users can issue virtual API keys, enforce budgets, and track usage, gaining full control over their AI infrastructure.

Why Use Otari and Key Benefits

Otari provides a robust solution for managing your LLM interactions with several compelling advantages:

One OpenAI and Anthropic-compatible endpoint for many providers.
Provider credentials stay securely behind the gateway.
Revocable API keys with user, workspace, and model scope.
Budget checks before spend and comprehensive usage records after settlement.
Local routing policies for failover, weighting, and learned selection.
Optional features including code execution, web search, multi-cloud provisioning (MCP), guardrails, and file understanding.

Installation

Getting started with Otari is straightforward. You can run an ephemeral standalone gateway using Docker:

docker run --rm -p 8000:8000 \
  -e OTARI_MASTER_KEY=SET_A_MASTER_KEY \
  -e OPENAI_API_KEY=YOUR_OPENAI_KEY \
  -e OTARI_CONFIG_YAML='default_pricing: true' \
  mzdotai/otari:latest \
  otari serve

For a full-stack deployment with persistent data, including PostgreSQL, clone the repository and use Docker Compose:

git clone https://github.com/mozilla-ai/otari
cd otari
cp config.example.yml config.yml
# Set a master key, provider credentials, and pricing in config.yml.
docker compose pull
docker compose up -d

The dashboard will be accessible at http://localhost:8000/. Optional services like code sandbox, web search, and guardrails can be enabled via Docker Compose profiles.

Examples

Once Otari is running, you can send requests using the generated bootstrap API key. OpenAI clients can also be configured by setting base_url to http://localhost:8000/v1.

Here's an example using curl to send a chat completion request:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer gw-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o-mini",
    "messages": [{"role": "user", "content": "Say hello."}]
  }'