otari: Run an OpenAI-Compatible LLM Gateway

Overview

Otari is a self-hostable, OpenAI-compatible LLM gateway built by Mozilla AI, with provider calls handled through its any-llm library. It sits between applications and model providers, giving teams one endpoint for requests while keeping provider credentials behind the gateway.

Use it as a standalone gateway, connect a data plane to otari.ai, or run it in a hybrid setup. Alongside request routing, it provides virtual keys, budget enforcement, and usage records to help teams manage access and spend across providers.

Key Features

• OpenAI- and Anthropic-compatible endpoints for chat completions, messages, and responses.
• Routes requests to more than 40 model providers through any-llm.
• Issues revocable API keys scoped to users, workspaces, and models.
• Checks budgets before dispatch and records usage after settlement.
• Supports routing policies for failover, weighting, and learned selection.
• Includes a dashboard for providers, models, keys, members, budgets, routing, and usage.
• Offers optional code execution, web search, MCP, inference guardrails, and file understanding.

Use Cases

• A team wants to switch among model providers without changing the endpoint configured in its applications.
• An organization needs provider credentials kept behind a gateway and wants to issue narrower, revocable keys to users or workspaces.
• Platform operators want budget checks and usage tracking across model requests.
• Developers want routing failover or weighting between configured providers.
• Teams already using otari.ai want to connect a gateway they operate as a data plane.

Getting Started

For an ephemeral Docker run, set a master key and provider credential:

docker run --rm -p 8000:8000 \
  -e OTARI_MASTER_KEY=SET_A_MASTER_KEY \
  -e OPENAI_API_KEY=YOUR_OPENAI_KEY \
  -e OTARI_CONFIG_YAML='default_pricing: true
providers:
  openai:
    api_key: ${OPENAI_API_KEY}
' \
  mzdotai/otari:latest otari serve

See the README for persistent deployment, configuration, and development instructions. The documentation covers gateway usage.

Alternatives

• a3m-router: A3M Router is a self-hosted multi-provider gateway focused on complexity-based routing, failover, caching, and ensembles rather than team keys and budgets.
• helicone: Helicone combines provider routing with request observability, traces, cost and latency tracking, and prompt management.

Considerations

• This is an operational service, not just a client library. Persistent deployment needs a database; the provided Docker Compose setup uses PostgreSQL, while the one-command Docker example stores SQLite data inside the container and loses it when the container stops.
• Python 3.13 or newer is required for direct development or installation. Docker and Compose are available deployment paths; the project does not list a Helm chart.
• Otari is young, despite frequent commits and releases. Its recent activity is strong, and contributions are not concentrated in one person, but teams should still evaluate its fit and deployment behavior before relying on it in production.
• The project is licensed under Apache-2.0. Review the license terms and any obligations relevant to your distribution or deployment.