Colibri: Run 744B GLM-5.2 MoE on Consumer Machines with Pure C

Introduction

Colibri is an impressive open-source project that redefines what's possible for running large language models on consumer hardware. It allows users to run the GLM-5.2 (744B-parameter Mixture-of-Experts) model on machines with as little as 25GB of RAM. The core innovation lies in its pure C engine, which has zero dependencies and efficiently streams model experts from disk, making an "immense model" accessible via a "tiny engine." This project is ideal for anyone looking to experiment with a frontier-class LLM without the need for high-end GPUs.

Installation

Getting Colibri up and running involves a few straightforward steps. First, clone the repository and build the engine. Python is only required for the one-time model conversion.

Clone the repository and build:
```
cd c
./setup.sh # Checks gcc/OpenMP, builds, self-tests
```
This command will build the coli executable and perform initial checks.

Install Python dependencies for conversion:

pip install torch safetensors huggingface_hub numpy

Convert the GLM-5.2 model:
```
./coli convert --model /path/to/your/model/directory
```
This process downloads the GLM-5.2-FP8 shards one by one, converts them to the int4 container format, and prepares the MTP head for speculative decoding. It's resumable and does not require the full 756 GB FP8 checkpoint to exist on disk at once. Alternatively, you can download a pre-converted int4 model from Hugging Face: https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4. Ensure you get the int8 MTP heads from the community clone if needed: https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp.

Examples

Once the model is converted and ready, you can start interacting with Colibri.

Interactive Chat:
To start an interactive chat session, simply point COLI_MODEL to your model directory:
```
COLI_MODEL=/path/to/your/model/directory ./coli chat
```
The engine automatically detects RAM budget, expert cache, and MTP settings.

OpenAI-compatible API Server:
Colibri also provides an OpenAI-compatible HTTP API server, allowing you to integrate the model into other applications.

cd c
COLI_MODEL=/path/to/your/model/directory COLI_API_KEY=local-secret ./coli serve \
  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri

You can then query it using curl:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Authorization: Bearer local-secret' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-colibri",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

Why Use It

Colibri stands out for several compelling reasons, making it a unique solution for large language model inference:

Accessibility: It democratizes access to frontier-class LLMs like GLM-5.2, allowing them to run on standard consumer machines, bypassing the need for expensive GPU clusters.
Efficiency: Built in pure C with zero dependencies, Colibri is incredibly lightweight and optimized. It intelligently streams experts from disk, ensuring that only necessary parts of the massive model are loaded into RAM, minimizing memory footprint.
Advanced Features: Colibri incorporates sophisticated techniques such as Native MTP speculative decoding for faster generation, compressed KV-cache persistence for warm conversation restarts, and a "learning cache" that automatically pins frequently used experts into RAM, making the engine faster with continued use.
Cross-Platform Support: It supports Linux, macOS, and native Windows 11 (via MinGW-w64), making it versatile for various development environments.
OpenAI-Compatible API: The built-in API server simplifies integration into existing AI applications and workflows.