{"name":"Mergoo: Efficiently Merge and Train Multiple LLM Experts","description":"Mergoo is an open-source Python library designed to simplify the merging of multiple Large Language Model (LLM) experts. It enables efficient training of these merged LLMs, allowing users to integrate knowledge from various generic or domain-specific models. The library supports several merging methods, including Mixture-of-Experts and Mixture-of-Adapters, across popular base models.","github":"https://github.com/Leeroo-AI/mergoo","url":"https://osrepos.com/repo/leeroo-ai-mergoo","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/leeroo-ai-mergoo","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/leeroo-ai-mergoo.md","json":"https://osrepos.com/repo/leeroo-ai-mergoo.json","topics":["artificial-intelligence","fine-tuning","generative-ai","large-language-models","llm","lora","mixture-of-experts","Python"],"keywords":["artificial-intelligence","fine-tuning","generative-ai","large-language-models","llm","lora","mixture-of-experts","Python"],"stars":null,"summary":"Mergoo is an open-source Python library designed to simplify the merging of multiple Large Language Model (LLM) experts. It enables efficient training of these merged LLMs, allowing users to integrate knowledge from various generic or domain-specific models. The library supports several merging methods, including Mixture-of-Experts and Mixture-of-Adapters, across popular base models.","content":"## Introduction\n\nMergoo is a powerful Python library developed by Leeroo-AI, aimed at streamlining the process of combining multiple Large Language Model (LLM) experts and efficiently training the resulting merged model. This library is particularly useful for integrating diverse knowledge bases from different LLM experts, whether they are generic or specialized for specific domains. Mergoo offers robust support for various merging techniques, including Mixture-of-Experts (MoE), Mixture-of-Adapters (MoA), and Layer-wise merging, providing flexibility in how models are combined.\n\nKey features of Mergoo include:\n\n*   **Multiple Merging Methods**: Supports Mixture-of-Experts, Mixture-of-Adapters, and Layer-wise merging.\n*   **Flexible Layer Merging**: Allows for flexible merging configurations for each layer.\n*   **Broad Base Model Support**: Compatible with popular models like Llama (including LLaMa3), Mistral, Phi3, and BERT.\n*   **Trainer Integration**: Works seamlessly with Hugging Face's `Trainer`, `SFTrainer`, and PEFT.\n*   **Device Compatibility**: Supports CPU, MPS, and GPU environments.\n*   **Training Choices**: Offers options to train only the Router of MoE layers or perform full fine-tuning of the merged LLM.\n\n## Installation\n\nGetting started with Mergoo is straightforward. You can install it using pip:\n\nbash\npip install mergoo\n\n\nFor the latest unstable version directly from GitHub:\n\nbash\npip install git+https://github.com/Leeroo-AI/mergoo\n\n\nAlternatively, you can install it from the source:\n\nbash\ngit clone https://github.com/Leeroo-AI/mergoo\ncd mergoo\npip install -e .\n\n\n## Examples\n\nMergoo simplifies the configuration and merging process. Here are examples for setting up configurations for different merging scenarios.\n\n### Fully Fine-tuned Experts\n\nThis configuration demonstrates merging fully fine-tuned LLM experts, such as combining math and code-specific Mistral models.\n\npython\nconfig = {\n    \"model_type\": \"mistral\",\n    \"num_experts_per_tok\": 2,\n    \"experts\": [\n        {\"expert_name\": \"base_expert\", \"model_id\": \"mistralai/Mistral-7B-v0.1\"},\n        {\"expert_name\": \"expert_1\", \"model_id\": \"meta-math/MetaMath-Mistral-7B\"},\n        {\"expert_name\": \"expert_2\", \"model_id\": \"ajibawa-2023/Code-Mistral-7B\"}\n    ],\n    \"router_layers\": [\"gate_proj\", \"up_proj\", \"down_proj\"]\n}\n\n\n### Mixture of Adapters (MoE on LoRA)\n\nMergoo can also build a routing layer on top of LoRAs, creating a Mixture of Adapters. This is ideal for merging LoRA fine-tuned LLM experts.\n\npython\nconfig = {\n    \"model_type\": \"mistral\",\n    \"num_experts_per_tok\": 2,\n    \"base_model\": \"mistralai/Mistral-7B-v0.1\",\n    \"experts\": [\n        {\"expert_name\": \"adapter_1\", \"model_id\": \"predibase/customer_support\"},\n        {\"expert_name\": \"adapter_2\", \"model_id\": \"predibase/customer_support_accounts\"},\n        {\"expert_name\": \"adapter_3\", \"model_id\": \"predibase/customer_support_orders\"},\n        {\"expert_name\": \"adapter_4\", \"model_id\": \"predibase/customer_support_payments\"}\n    ],\n}\n\n\nAfter configuring, you can merge and save the checkpoint:\n\npython\nimport torch\nfrom mergoo.compose_experts import ComposeExperts\n\nmodel_id = \"data/mistral_lora_moe\"\nexpertmerger = ComposeExperts(config, torch_dtype=torch.float16)\nexpertmerger.compose()\nexpertmerger.save_checkpoint(model_id)\n\n\nThen, load and fine-tune the merged model using Hugging Face Trainer:\n\npython\nfrom transformers import Trainer\nfrom mergoo.models.modeling_mistral import MistralForCausalLM\n\nmodel = MistralForCausalLM.from_pretrained(\"data/mistral_lora_moe\") \n# NOTE: 'gate' / router layers are untrained hence weight loading warning would appeare for them\n\ntrainer = Trainer( ... )\ntrainer.train()\n\n\n## Why Use Mergoo?\n\nMergoo addresses the growing need to combine specialized LLMs effectively. By providing a flexible and efficient framework for merging, it allows developers and researchers to:\n\n*   **Leverage Diverse Expertise**: Integrate knowledge from multiple domain-specific or task-specific LLMs into a single, more capable model.\n*   **Improve Model Performance**: Potentially enhance performance on complex tasks by routing inputs to the most relevant expert or combining their strengths.\n*   **Reduce Training Costs**: Efficiently train merged models, focusing on router layers or specific components, rather than retraining entire large models from scratch.\n*   **Simplify Complex Workflows**: Abstract away the complexities of implementing Mixture-of-Experts or Mixture-of-Adapters architectures.\n*   **Stay Current**: Support for the latest LLMs like Llama3 and integration with Hugging Face ecosystem ensures compatibility with modern AI practices.\n\n## Links\n\nExplore Mergoo further through these official resources:\n\n*   **GitHub Repository**: [https://github.com/Leeroo-AI/mergoo](https://github.com/Leeroo-AI/mergoo){:target=\"_blank\"}\n*   **MoE with fully fine-tuned LLM experts Notebook**: [https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/llama_compose_trainer.ipynb](https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/llama_compose_trainer.ipynb){:target=\"_blank\"}\n*   **MoE with LoRA fine-tuned experts Notebook**: [https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/Mistral_lora_compose_trainer.ipynb](https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/Mistral_lora_compose_trainer.ipynb){:target=\"_blank\"}\n*   **LLaMa3-based Experts Notebook**: [https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/integrate_llama3_experts.ipynb](https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/integrate_llama3_experts.ipynb){:target=\"_blank\"}\n*   **Phi3-based Experts Notebook**: [https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/integrate_phi3_experts.ipynb](https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/integrate_phi3_experts.ipynb){:target=\"_blank\"}\n*   **Hugging Face Blog Post**: [https://huggingface.co/blog/alirezamsh/mergoo](https://huggingface.co/blog/alirezamsh/mergoo){:target=\"_blank\"}\n*   **Leeroo AI Website**: [https://www.leeroo.com](https://www.leeroo.com){:target=\"_blank\"}\n*   **Leeroo AI Twitter**: [https://twitter.com/LeerooAI](https://twitter.com/LeerooAI){:target=\"_blank\"}\n*   **Leeroo AI LinkedIn**: [https://www.linkedin.com/company/leeroo](https://www.linkedin.com/company/leeroo){:target=\"_blank\"}\n*   **Leeroo AI Discord**: [https://discord.gg/hqVbPNNEZM](https://discord.gg/hqVbPNNEZM){:target=\"_blank\"}","metrics":{"detailViews":1,"githubClicks":1},"dates":{"published":null,"modified":"2026-07-07T12:26:56.000Z"}}