a3m-router: Route LLM Requests Across Models

Overview

A3M Router is a self-hosted, OpenAI-compatible gateway for routing LLM requests across providers. Its defining routing option is Jev System One, exposed as model="jev-auto"; the project also offers a heuristic model="auto" mode. The goal is to match requests to a suitable provider rather than send every prompt to the same expensive model.

The project is aimed at developers who want to run a routing service on their own infrastructure and keep using the OpenAI chat completions interface. It is a young, rapidly changing TypeScript project, so its claims about savings and answer quality should be checked against your own workload and provider setup.

Key Features

• Provides an OpenAI-compatible chat completions API and a models endpoint.
• Offers heuristic auto routing and Jev System One jev-auto routing.
• Advertises support for more than 80 providers, with availability checked at runtime.
• Includes failover behavior for provider availability problems.
• Supports semantic caching and conversation memory options.
• Can run parallel provider calls and combine results using ensemble strategies.
• Includes CLI commands for serving, previewing a route, checking provider health, and benchmarking.
• Can be run with Node.js or Docker.

Use Cases

• An application team wants to keep its existing OpenAI client integration while routing requests through a self-hosted gateway.
• A developer handling mixed workloads wants simple prompts routed differently from complex reasoning requests.
• A team wants fallback providers when its preferred model service is unavailable.
• Developers evaluating model responses want to compare multiple providers on the same request using parallel ensembles.

Getting Started

npm install adaptive-memory-multi-model-router
npx a3m-router serve

Then point an OpenAI client at http://localhost:8787/v1. See the README for configuration and usage details.

Alternatives

• otari: Otari adds virtual keys, budgets, and team usage controls, while a3m-router focuses on Jev System One task-based single-pass routing and ensembles.
• TurboLLM: TurboLLM focuses on serving local models with hardware-aware inference, while a3m-router routes requests among models and providers.
• OmniRoute: OmniRoute emphasizes configurable, quota-aware fallback and token compression, while a3m-router uses task-based Jev System One routing and optional ensembles.
• freellmapi: FreeLLMAPI combines free provider accounts and quota tracking for personal use, while a3m-router routes by task and can run parallel model ensembles.

Considerations

The repository is less than six months old and is changing quickly. Recent activity includes frequent commits and releases, but nearly all recorded contributions come from one of its two contributors, which creates a continuity risk. Issue resolution has taken longer than pull-request merging on median, so do not assume that open issues will be handled quickly.

Routing quality, provider pricing, and availability depend on the workload and external providers; the README's savings examples are project claims, not a guarantee for your traffic. The README also notes that this is not suitable if you require the same premium model for every request or cannot run a local service. The project is MIT-licensed; review the license and the terms of any model providers you connect before deployment.