RouteLLM: Route LLM Requests to Balance Cost and Quality

Overview

RouteLLM helps teams decide which of two language models should answer each prompt. Rather than sending every request to the strongest, most expensive model, it uses a router to direct simpler prompts to a cheaper model and reserve the stronger one for prompts that need it.

It is intended for developers and researchers who want to serve, calibrate, or compare LLM routing strategies. It includes pretrained routers and an evaluation framework, but routing thresholds and results depend on the query mix and model pair in use.

Key Features

• Python controller designed as a drop-in replacement for the OpenAI client.
• OpenAI-compatible server for routing requests from existing clients.
• Five router options: matrix factorization, weighted ranking, BERT, a causal LLM classifier, and random routing.
• Configurable strong and weak model choices, with LiteLLM support for multiple providers and OpenAI-compatible endpoints.
• Threshold calibration to control the proportion of requests sent to the stronger model.
• Evaluation support for MMLU, GSM8K, and MT Bench, with cached results.
• Extension points for adding routers and benchmarks.

Use Cases

• Application teams handling many requests can route simpler prompts to a lower-cost model while preserving access to a stronger model for harder queries.
• Developers integrating through the OpenAI client interface can test routing without replacing their entire client workflow.
• ML researchers can compare routing strategies on supported benchmarks or implement a new router.
• Teams adapting routing to their traffic can calibrate thresholds with a representative sample of incoming queries.

Getting Started

Install the serving and evaluation extras:

pip install "routellm[serve,eval]"

See the README for model setup, server configuration, and usage details.

Considerations

• You need credentials and access for the selected model providers. The README also notes that an OpenAI API key is currently required for embeddings used by the mf and sw_ranking routers.
• Routing quality and the share of requests sent to each model depend on the model pair, incoming prompts, and threshold. Calibrate with data representative of your actual traffic.
• The supplied trained routers are based on a GPT-4 and Mixtral model pair. The README reports that they generalize to other pairs, but results for a different deployment should be evaluated.
• The repository metadata lists its last push as 2024-08-10, so check the repository for current compatibility and maintenance status before adopting it.