# Colibri: Run 744B GLM-5.2 MoE on Consumer Machines with Pure C

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/justvugg-colibri
Generated for open source discovery and AI-assisted research.

Colibri is an innovative project that enables running the massive 744B-parameter GLM-5.2 Mixture-of-Experts (MoE) model on consumer-grade machines with as little as 25GB of RAM. It achieves this remarkable feat through a pure C engine with zero dependencies, streaming model experts from disk on demand. This allows users to interact with a frontier-class LLM without requiring expensive GPU hardware.

GitHub: https://github.com/JustVugg/colibri
OSRepos URL: https://osrepos.com/repo/justvugg-colibri

## Summary

Colibri is an innovative project that enables running the massive 744B-parameter GLM-5.2 Mixture-of-Experts (MoE) model on consumer-grade machines with as little as 25GB of RAM. It achieves this remarkable feat through a pure C engine with zero dependencies, streaming model experts from disk on demand. This allows users to interact with a frontier-class LLM without requiring expensive GPU hardware.

## Topics

- C
- LLM
- AI
- Machine Learning
- MoE
- GitHub
- Open Source
- Performance

## Repository Information

Last analyzed by OSRepos: Sat Jul 11 2026 21:46:28 GMT+0100 (Western European Summer Time)
Detail views: 3
GitHub clicks: 1

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction
Colibri is an impressive open-source project that redefines what's possible for running large language models on consumer hardware. It allows users to run the GLM-5.2 (744B-parameter Mixture-of-Experts) model on machines with as little as 25GB of RAM. The core innovation lies in its pure C engine, which has zero dependencies and efficiently streams model experts from disk, making an "immense model" accessible via a "tiny engine." This project is ideal for anyone looking to experiment with a frontier-class LLM without the need for high-end GPUs.

## Installation
Getting Colibri up and running involves a few straightforward steps. First, clone the repository and build the engine. Python is only required for the one-time model conversion.

1.  **Clone the repository and build:**
    bash
    cd c
    ./setup.sh # Checks gcc/OpenMP, builds, self-tests
    
    This command will build the `coli` executable and perform initial checks.

2.  **Install Python dependencies for conversion:**
    bash
    pip install torch safetensors huggingface_hub numpy
    

3.  **Convert the GLM-5.2 model:**
    bash
    ./coli convert --model /path/to/your/model/directory
    
    This process downloads the GLM-5.2-FP8 shards one by one, converts them to the int4 container format, and prepares the MTP head for speculative decoding. It's resumable and does not require the full 756 GB FP8 checkpoint to exist on disk at once. Alternatively, you can download a pre-converted int4 model from Hugging Face: [https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4). Ensure you get the int8 MTP heads from the community clone if needed: [https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp).

## Examples
Once the model is converted and ready, you can start interacting with Colibri.

1.  **Interactive Chat:**
    To start an interactive chat session, simply point `COLI_MODEL` to your model directory:
    bash
    COLI_MODEL=/path/to/your/model/directory ./coli chat
    
    The engine automatically detects RAM budget, expert cache, and MTP settings.

2.  **OpenAI-compatible API Server:**
    Colibri also provides an OpenAI-compatible HTTP API server, allowing you to integrate the model into other applications.
    bash
    cd c
    COLI_MODEL=/path/to/your/model/directory COLI_API_KEY=local-secret ./coli serve \
      --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
    
    You can then query it using `curl`:
    bash
    curl http://127.0.0.1:8000/v1/chat/completions \
      -H 'Authorization: Bearer local-secret' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "glm-5.2-colibri",
        "messages": [{"role": "user", "content": "Hello"}],
        "stream": true
      }'
    

## Why Use It
Colibri stands out for several compelling reasons, making it a unique solution for large language model inference:

*   **Accessibility:** It democratizes access to frontier-class LLMs like GLM-5.2, allowing them to run on standard consumer machines, bypassing the need for expensive GPU clusters.
*   **Efficiency:** Built in pure C with zero dependencies, Colibri is incredibly lightweight and optimized. It intelligently streams experts from disk, ensuring that only necessary parts of the massive model are loaded into RAM, minimizing memory footprint.
*   **Advanced Features:** Colibri incorporates sophisticated techniques such as Native MTP speculative decoding for faster generation, compressed KV-cache persistence for warm conversation restarts, and a "learning cache" that automatically pins frequently used experts into RAM, making the engine faster with continued use.
*   **Cross-Platform Support:** It supports Linux, macOS, and native Windows 11 (via MinGW-w64), making it versatile for various development environments.
*   **OpenAI-Compatible API:** The built-in API server simplifies integration into existing AI applications and workflows.

## Links
*   **GitHub Repository:** [https://github.com/JustVugg/colibri](https://github.com/JustVugg/colibri)
*   **Hugging Face Model (int4):** [https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4)
*   **Hugging Face Model (int8 MTP):** [https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp)