{"name":"llm-consortium: Orchestrating Multiple LLMs for Consensus and Refinement","description":"llm-consortium is a powerful plugin for the `llm` package, designed to enhance problem-solving by orchestrating multiple large language models. It implements a parallel reasoning method that iteratively refines responses and achieves consensus through structured dialogue, evaluation, and arbitration. This system leverages the collective intelligence of diverse LLMs to tackle complex problems more effectively.","github":"https://github.com/irthomasthomas/llm-consortium","url":"https://osrepos.com/repo/irthomasthomas-llm-consortium","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/irthomasthomas-llm-consortium","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/irthomasthomas-llm-consortium.md","json":"https://osrepos.com/repo/irthomasthomas-llm-consortium.json","topics":["ai","llms","Python","artificial intelligence","large language models","llm orchestration","natural language processing","open source"],"keywords":["ai","llms","Python","artificial intelligence","large language models","llm orchestration","natural language processing","open source"],"stars":null,"summary":"llm-consortium is a powerful plugin for the `llm` package, designed to enhance problem-solving by orchestrating multiple large language models. It implements a parallel reasoning method that iteratively refines responses and achieves consensus through structured dialogue, evaluation, and arbitration. This system leverages the collective intelligence of diverse LLMs to tackle complex problems more effectively.","content":"## Introduction\n\n`llm-consortium` is an innovative plugin for the `llm` package that introduces a model consortium system for advanced problem-solving. Inspired by Andrej Karpathy's observation that optimal performance often comes from asking all available models and having them reach a consensus, this tool orchestrates multiple diverse language models to collaboratively solve complex problems.\n\nThe core idea revolves around parallel reasoning, where various LLMs contribute to a solution. The system then iteratively refines these responses, synthesizes them, and uses a designated arbiter model to evaluate and arbitrate until a high-confidence consensus is achieved.\n\n## Installation\n\nFirst, ensure you have the `llm` package installed. You can install it using `uv` or `pipx`:\n\nUsing `uv`:\nbash\nuv tool install llm\n\n\nUsing `pipx`:\nbash\npipx install llm\n\n\nOnce `llm` is installed, you can install the `llm-consortium` plugin:\n\nbash\nllm install llm-consortium\n\n\n## Examples\n\n### Command Line Usage\n\nThe `consortium` command defaults to the `run` subcommand for concise usage.\n\n**Basic Usage:**\nbash\nllm consortium \"What are the key considerations for AGI safety?\"\n\n\nOr, if you have saved a consortium model (e.g., named `my-consortium`):\nbash\nllm -m my-consortium \"What are the key considerations for AGI safety?\"\n\n\nThis command will send your prompt to multiple models in parallel, gather responses, use an arbiter model to synthesize them, and iterate to refine the answer until a specified confidence threshold or maximum iteration count is reached.\n\n**Conversation Continuation Usage (New in v0.3.2):**\n\nTo continue the most recent conversation:\nbash\n# Initial prompt\nllm -m my-consortium \"Tell me about the planet Mars.\"\n# Follow-up\nllm -c \"How long does it take to get there?\"\n\n\nTo continue a specific conversation using its ID:\nbash\n# Initial prompt (note the conversation ID, e.g., 01jscjy50ty4ycsypbq6h4ywhh)\nllm -m my-consortium \"Tell me about Jupiter.\"\n\n# Follow-up using the ID\nllm -c --cid 01jscjy50ty4ycsypbq6h4ywhh \"What are its major moons?\"\n\n\n**Advanced Example:**\nYou can specify individual instance counts for models, a custom arbiter, and adjust thresholds:\nbash\nllm consortium \"Your complex query\" \\\n  -m o3-mini:1 \\\n  -m gpt-4o:2 \\\n  -m gemini-2:3 \\\n  --arbiter gemini-2 \\\n  --confidence-threshold 1 \\\n  --max-iterations 4 \\\n  --min-iterations 3 \\\n  --output results.json\n\n\n### Managing Consortium Configurations\n\nYou can save a consortium configuration as a named model for reuse:\nbash\nllm consortium save my-consortium \\\n    --model claude-3-opus-20240229 \\\n    --model gpt-4 \\\n    --arbiter claude-3-opus-20240229 \\\n    --confidence-threshold 0.9 \\\n    --max-iterations 5 \\\n    --min-iterations 1 \\\n    --system \"Your custom system prompt\"\n\n\nOnce saved, invoke your custom consortium:\nbash\nllm -m my-consortium \"What are the key considerations for AGI safety?\"\n\n\n### Programmatic Usage\n\nIntegrate `llm-consortium` into your Python code using the `create_consortium` helper:\n\npython\nfrom llm_consortium import create_consortium\n\norchestrator = create_consortium(\n    models=[\"o3-mini:1\", \"gpt-4o:2\", \"gemini-2:3\"],\n    confidence_threshold=1,\n    max_iterations=4,\n    min_iterations=3,\n    arbiter=\"gemini-2\",\n    raw=True\n)\n\nresult = orchestrator.orchestrate(\"Your prompt here\")\nprint(f\"Synthesized Response: {result['synthesis']['synthesis']}\")\n\n\n## Why Use llm-consortium?\n\n`llm-consortium` offers several compelling features for advanced LLM applications:\n\n*   **Multi-Model Orchestration**: Coordinate responses from multiple models in parallel, leveraging diverse perspectives.\n*   **Iterative Refinement**: Automatically refine output until a desired confidence threshold is achieved, improving answer quality.\n*   **Advanced Arbitration**: Utilizes a designated arbiter model to synthesize and critically evaluate responses, ensuring robust consensus.\n*   **Database Logging**: All interactions are logged to SQLite, providing a clear audit trail and debugging capabilities.\n*   **Configurable Parameters**: Adjustable confidence thresholds, iteration limits, and model selection offer fine-grained control.\n*   **Flexible Model Instance Counts**: Specify individual instance counts per model, allowing for optimized resource allocation.\n*   **Conversation Continuation**: Seamlessly continue previous conversations, enhancing user experience for multi-turn interactions.\n\n## Links\n\n*   **GitHub Repository**: [https://github.com/irthomasthomas/llm-consortium](https://github.com/irthomasthomas/llm-consortium){:target=\"_blank\"}\n*   **`llm` package**: [https://github.com/simonw/llm](https://github.com/simonw/llm){:target=\"_blank\"}","metrics":{"detailViews":4,"githubClicks":6},"dates":{"published":null,"modified":"2025-12-01T00:01:26.000Z"}}