A3M Router: Biology-Inspired LLM Routing for Cost Savings and Performance

Introduction

A3M Router is an open-source, biology-inspired LLM router that intelligently directs queries across more than 80 AI providers. It addresses the challenges of centralized AI routing and high costs, offering a decentralized and highly efficient alternative. Inspired by 3 billion years of biological intelligence, A3M Router aims to build the "nervous system" for AI, ensuring optimal performance and significant cost savings.

Why Use A3M Router and Key Benefits

A3M Router stands out by offering substantial advantages for developers and organizations working with LLMs. Its core philosophy, inspired by biological intelligence, translates into tangible benefits:

• Significant Cost Reduction: By intelligently routing queries, A3M Router can reduce AI costs by 70-95%. Simple questions, for example, can be routed to cheaper models like Groq, costing 300x less than premium models.
• Superior Performance: Benchmarks show A3M Router achieving a P99 latency of 162ms, making it 14% faster than some alternatives.
• Exceptional Uptime: Leveraging "Swarm Intelligence," similar to how ant colonies find optimal paths, the router ensures 99.99% uptime. If a model fails, requests automatically avoid it, and new paths emerge.
• Adaptive Learning: Inspired by "Neural Plasticity," A3M Router constantly adapts. Time-decayed weights ensure that recent provider performance is prioritized, preventing overfitting to outdated data.
• Provider Diversity: The "Competitive Exclusion" principle, using the EXP3 algorithm, promotes a natural equilibrium among providers, preventing any single one from dominating and ensuring a diverse ecosystem.
• Intelligent Resource Allocation: The "Handicap Principle" treats cost as a credibility signal. High-cost models are reserved for high-stakes, complex queries, ensuring optimal resource use.
• Extensive Provider Coverage: With support for over 80 providers, including OpenAI, Anthropic, Google, Groq, and Mistral, A3M Router offers unparalleled flexibility.
• Open-Source and Self-Hostable: Unlike some centralized solutions, A3M Router is 100% open-source and can be self-hosted, giving users full control and transparency.

Installation

Getting started with A3M Router is straightforward. Choose your preferred package manager:

# npm
npm install adaptive-memory-multi-model-router
npx a3m-router serve

# Python
pip install a3m-router
python -m a3m_router.serve

# Docker
docker run -p 8787:8787 ghcr.io/das-rebel/a3m-router

Examples

Once installed, you can integrate A3M Router into your applications. It's designed to be compatible with OpenAI's API format.

Basic Routing

Simply change your base_url and set model="auto":

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8787/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="auto", # A3M picks the best provider
    messages=[{"role": "user", "content": "What is Python?"}]
)
print(response.choices[0].message.content)

Parallel Ensemble, Best Answer Mode

For critical queries where the best possible answer is paramount, A3M Router can query multiple providers simultaneously:

from a3m.router import A3MRouter

router = A3MRouter(
    model="auto",
    parallel_ensemble=3, # Call 3 providers simultaneously
)

result = router.route(
    messages=[{"role": "user", "content": "Explain quantum entanglement"}],
    ensemble_timeout_ms=10000,
)

print(f"Best from: {result.provider}")
print(f"Response: {result.content}")

Semantic Memory and Context

A3M Router also includes semantic memory capabilities to maintain context across interactions:

from a3m.router import A3MRouter

router = A3MRouter(
    model="auto",
    memory={
        "type": "semantic",
        "window": 10, # Last 10 exchanges
        "similarity_threshold": 0.85,
    }
)

# Second call uses cached context automatically
result = router.route(
    messages=[{"role": "user", "content": "What framework should I use?"}]
)
# A3M knows "Python web app" from previous context

Links

Explore A3M Router further through its official channels:

• GitHub Repository
• npm Package
• PyPI Package
• Twitter