irthomasthomas

llm-consortium: Coordinate Multiple LLMs to Refine Answers

llm-consortium is a Python plugin for the llm package that runs multiple language models in parallel, then uses an arbiter to evaluate and synthesize their responses. It suits users who want collaborative model reasoning and configurable consensus workflows.

PythonApache-2.0AILLMCLI
View on GitHub Compare
Stars
407
Forks
42
Stars, last 30 days
+1

Our take

Health score

23/100

How it is scored
Activity
37
Community
12
Issues
21
Pull requests
20

+1 stars in the last 30 days

  • Fewer contributors than 86% of the projects we track

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

409

Forks

42

Change since Nov 16, 2026: 0 stars·0 forks

Overview

llm-consortium extends the llm command-line ecosystem with a way to ask several language models the same question, compare their responses, and have an arbiter synthesize or rank them. It targets problems where comparing candidate answers may improve the result over relying on a single model.

The workflow can repeat with additional refinement rounds, stopping when its confidence threshold is met or its iteration limit is reached. It is useful when you can access multiple models and want a configurable process for combining their output, rather than a guarantee that consensus is correct.

Key Features

  • Sends prompts to multiple model instances in parallel, with per-model instance counts.
  • Uses a designated arbiter to synthesize responses or rank them using configurable judging methods.
  • Supports iterative refinement, confidence thresholds, and iteration limits.
  • Offers semantic filtering that clusters response embeddings before arbitration.
  • Logs interactions and response metadata in SQLite, with optional embedding visualization.
  • Saves reusable consortium configurations and supports continuing conversations through the llm CLI.
  • Provides a Python create_consortium helper for programmatic orchestration.

Use Cases

  • Researchers can compare model-generated explanations or candidate solutions when a task benefits from multiple perspectives.
  • Developers can create a reusable command-line workflow that combines models and an arbiter for recurring questions.
  • Teams can experiment with ranking, synthesis, or semantic filtering strategies to examine how different arbitration approaches affect responses.
  • Python developers can embed multi-model orchestration in scripts that already use the llm ecosystem.

Getting Started

Install the plugin with the llm CLI:

llm install llm-consortium

Then save a consortium configuration and invoke it as an llm model. See the README for configuration examples, optional features, and setup details.

Considerations

This is an orchestration plugin, not a language model: you need compatible model providers and access to the models you configure. Embedding-based features have optional dependencies and may require provider credentials. Agreement or geometric confidence reflects similarity between responses, not factual correctness, so outputs still need appropriate verification.

Found this useful?

Share it with someone who would like llm-consortium.

Comparisons