{"name":"Colibri: Run 744B GLM-5.2 MoE on Consumer Machines with Pure C","description":"Colibri is an innovative project that enables running the massive 744B-parameter GLM-5.2 Mixture-of-Experts (MoE) model on consumer-grade machines with as little as 25GB of RAM. It achieves this remarkable feat through a pure C engine with zero dependencies, streaming model experts from disk on demand. This allows users to interact with a frontier-class LLM without requiring expensive GPU hardware.","github":"https://github.com/JustVugg/colibri","url":"https://osrepos.com/repo/justvugg-colibri","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/justvugg-colibri","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/justvugg-colibri.md","json":"https://osrepos.com/repo/justvugg-colibri.json","topics":["C","LLM","AI","Machine Learning","MoE","GitHub","Open Source","Performance"],"keywords":["C","LLM","AI","Machine Learning","MoE","GitHub","Open Source","Performance"],"stars":null,"summary":"Colibri is an innovative project that enables running the massive 744B-parameter GLM-5.2 Mixture-of-Experts (MoE) model on consumer-grade machines with as little as 25GB of RAM. It achieves this remarkable feat through a pure C engine with zero dependencies, streaming model experts from disk on demand. This allows users to interact with a frontier-class LLM without requiring expensive GPU hardware.","content":"## Introduction\nColibri is an impressive open-source project that redefines what's possible for running large language models on consumer hardware. It allows users to run the GLM-5.2 (744B-parameter Mixture-of-Experts) model on machines with as little as 25GB of RAM. The core innovation lies in its pure C engine, which has zero dependencies and efficiently streams model experts from disk, making an \"immense model\" accessible via a \"tiny engine.\" This project is ideal for anyone looking to experiment with a frontier-class LLM without the need for high-end GPUs.\n\n## Installation\nGetting Colibri up and running involves a few straightforward steps. First, clone the repository and build the engine. Python is only required for the one-time model conversion.\n\n1.  **Clone the repository and build:**\n    bash\n    cd c\n    ./setup.sh # Checks gcc/OpenMP, builds, self-tests\n    \n    This command will build the `coli` executable and perform initial checks.\n\n2.  **Install Python dependencies for conversion:**\n    bash\n    pip install torch safetensors huggingface_hub numpy\n    \n\n3.  **Convert the GLM-5.2 model:**\n    bash\n    ./coli convert --model /path/to/your/model/directory\n    \n    This process downloads the GLM-5.2-FP8 shards one by one, converts them to the int4 container format, and prepares the MTP head for speculative decoding. It's resumable and does not require the full 756 GB FP8 checkpoint to exist on disk at once. Alternatively, you can download a pre-converted int4 model from Hugging Face: [https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4). Ensure you get the int8 MTP heads from the community clone if needed: [https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp).\n\n## Examples\nOnce the model is converted and ready, you can start interacting with Colibri.\n\n1.  **Interactive Chat:**\n    To start an interactive chat session, simply point `COLI_MODEL` to your model directory:\n    bash\n    COLI_MODEL=/path/to/your/model/directory ./coli chat\n    \n    The engine automatically detects RAM budget, expert cache, and MTP settings.\n\n2.  **OpenAI-compatible API Server:**\n    Colibri also provides an OpenAI-compatible HTTP API server, allowing you to integrate the model into other applications.\n    bash\n    cd c\n    COLI_MODEL=/path/to/your/model/directory COLI_API_KEY=local-secret ./coli serve \\\n      --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri\n    \n    You can then query it using `curl`:\n    bash\n    curl http://127.0.0.1:8000/v1/chat/completions \\\n      -H 'Authorization: Bearer local-secret' \\\n      -H 'Content-Type: application/json' \\\n      -d '{\n        \"model\": \"glm-5.2-colibri\",\n        \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n        \"stream\": true\n      }'\n    \n\n## Why Use It\nColibri stands out for several compelling reasons, making it a unique solution for large language model inference:\n\n*   **Accessibility:** It democratizes access to frontier-class LLMs like GLM-5.2, allowing them to run on standard consumer machines, bypassing the need for expensive GPU clusters.\n*   **Efficiency:** Built in pure C with zero dependencies, Colibri is incredibly lightweight and optimized. It intelligently streams experts from disk, ensuring that only necessary parts of the massive model are loaded into RAM, minimizing memory footprint.\n*   **Advanced Features:** Colibri incorporates sophisticated techniques such as Native MTP speculative decoding for faster generation, compressed KV-cache persistence for warm conversation restarts, and a \"learning cache\" that automatically pins frequently used experts into RAM, making the engine faster with continued use.\n*   **Cross-Platform Support:** It supports Linux, macOS, and native Windows 11 (via MinGW-w64), making it versatile for various development environments.\n*   **OpenAI-Compatible API:** The built-in API server simplifies integration into existing AI applications and workflows.\n\n## Links\n*   **GitHub Repository:** [https://github.com/JustVugg/colibri](https://github.com/JustVugg/colibri)\n*   **Hugging Face Model (int4):** [https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4)\n*   **Hugging Face Model (int8 MTP):** [https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp)","metrics":{"detailViews":3,"githubClicks":1},"dates":{"published":null,"modified":"2026-07-11T20:46:28.000Z"}}