llm-consortium: Orchestrating Multiple LLMs for Consensus and Refinement

Introduction

llm-consortium is an innovative plugin for the llm package that introduces a model consortium system for advanced problem-solving. Inspired by Andrej Karpathy's observation that optimal performance often comes from asking all available models and having them reach a consensus, this tool orchestrates multiple diverse language models to collaboratively solve complex problems.

The core idea revolves around parallel reasoning, where various LLMs contribute to a solution. The system then iteratively refines these responses, synthesizes them, and uses a designated arbiter model to evaluate and arbitrate until a high-confidence consensus is achieved.

Installation

First, ensure you have the llm package installed. You can install it using uv or pipx:

Using uv:

uv tool install llm

Using pipx:

pipx install llm

Once llm is installed, you can install the llm-consortium plugin:

llm install llm-consortium

Examples

Command Line Usage

The consortium command defaults to the run subcommand for concise usage.

Basic Usage:

llm consortium "What are the key considerations for AGI safety?"

Or, if you have saved a consortium model (e.g., named my-consortium):

llm -m my-consortium "What are the key considerations for AGI safety?"

This command will send your prompt to multiple models in parallel, gather responses, use an arbiter model to synthesize them, and iterate to refine the answer until a specified confidence threshold or maximum iteration count is reached.

Conversation Continuation Usage (New in v0.3.2):

To continue the most recent conversation:

# Initial prompt
llm -m my-consortium "Tell me about the planet Mars."
# Follow-up
llm -c "How long does it take to get there?"

To continue a specific conversation using its ID:

# Initial prompt (note the conversation ID, e.g., 01jscjy50ty4ycsypbq6h4ywhh)
llm -m my-consortium "Tell me about Jupiter."

# Follow-up using the ID
llm -c --cid 01jscjy50ty4ycsypbq6h4ywhh "What are its major moons?"

Advanced Example:
You can specify individual instance counts for models, a custom arbiter, and adjust thresholds:

llm consortium "Your complex query" \
  -m o3-mini:1 \
  -m gpt-4o:2 \
  -m gemini-2:3 \
  --arbiter gemini-2 \
  --confidence-threshold 1 \
  --max-iterations 4 \
  --min-iterations 3 \
  --output results.json

Managing Consortium Configurations

You can save a consortium configuration as a named model for reuse:

llm consortium save my-consortium \
    --model claude-3-opus-20240229 \
    --model gpt-4 \
    --arbiter claude-3-opus-20240229 \
    --confidence-threshold 0.9 \
    --max-iterations 5 \
    --min-iterations 1 \
    --system "Your custom system prompt"

Once saved, invoke your custom consortium:

llm -m my-consortium "What are the key considerations for AGI safety?"

Programmatic Usage

Integrate llm-consortium into your Python code using the create_consortium helper:

from llm_consortium import create_consortium

orchestrator = create_consortium(
    models=["o3-mini:1", "gpt-4o:2", "gemini-2:3"],
    confidence_threshold=1,
    max_iterations=4,
    min_iterations=3,
    arbiter="gemini-2",
    raw=True
)

result = orchestrator.orchestrate("Your prompt here")
print(f"Synthesized Response: {result['synthesis']['synthesis']}")

Why Use llm-consortium?

llm-consortium offers several compelling features for advanced LLM applications:

Multi-Model Orchestration: Coordinate responses from multiple models in parallel, leveraging diverse perspectives.
Iterative Refinement: Automatically refine output until a desired confidence threshold is achieved, improving answer quality.
Advanced Arbitration: Utilizes a designated arbiter model to synthesize and critically evaluate responses, ensuring robust consensus.
Database Logging: All interactions are logged to SQLite, providing a clear audit trail and debugging capabilities.
Configurable Parameters: Adjustable confidence thresholds, iteration limits, and model selection offer fine-grained control.
Flexible Model Instance Counts: Specify individual instance counts per model, allowing for optimized resource allocation.
Conversation Continuation: Seamlessly continue previous conversations, enhancing user experience for multi-turn interactions.