Cactus: Cross-Platform AI Inference Engine for Mobile Devices

Introduction

Cactus is a powerful open-source project by Cactus Compute, Inc. and UCLA's BruinAI, offering an advanced AI inference engine optimized for mobile devices. It provides a comprehensive stack, from low-level hardware-specific kernels to a high-level API, enabling efficient execution of AI models on various platforms like iOS and Android.

The architecture of Cactus is layered, starting with Cactus Kernels for ARM-specific SIMD operations, building up to Cactus Graph for unified computation, Cactus Models for implementing SOTA models, Cactus Engine for transformer inference with NPU support, and finally Cactus FFI for an OpenAI-compatible C API.

Installation

Getting started with Cactus on a Mac is straightforward. First, clone the repository and navigate into its directory. Then, source the setup script to configure your environment.

git clone https://github.com/cactus-compute/cactus && cd cactus && source ./setup

Remember to run source ./setup in any new terminal session. You can use cactus run [model] to open a playground, cactus download [model] to get models, and cactus build to compile for ARM targets like Apple or Android.

Examples

Cactus provides intuitive APIs for both graph-level operations and high-level model inference. Below are examples demonstrating how to use Cactus Graph & Kernel for custom computations and Cactus Engine & FFI for running LLM inference.

Cactus Graph & Kernel

This example shows how to define a computation graph, set inputs, execute it, and retrieve results using Cactus's low-level graph API.

#include cactus.h

CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);

auto x1 = graph.matmul(a, b, false);
auto x2 = graph.transpose(x1);
auto result = graph.matmul(b, x2, true);

float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f};
float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};

graph.set_input(a, a_data, Precision::FP16);
graph.set_input(b, b_data, Precision::INT8);

graph.execute();
void* output_data = graph.get_output(result);

graph.hard_reset();

Cactus Engine & FFI

This snippet demonstrates using the Cactus Engine via its FFI to perform AI inference, similar to OpenAI's API, with support for chat messages and generation options.

#include cactus.h

cactus_set_pro_key("");                  // email founders@cactuscompute.com for optional key

cactus_model_t model = cactus_init(
    "path/to/weight/folder",             // section to generate weigths below
    "txt/or/md/file/or/dir/with/many",   // nullptr if none, cactus does automatic fast RAG
);

const char* messages = R"([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Henry Ndubuaku"}
])";

const char* options = R"({
    "max_tokens": 50,
    "stop_sequences": ["<|im_end|>"]
})";

char response[4096];
int result = cactus_complete(
    model,                               // model handle from cactus_init
    messages,                            // JSON array of chat messages
    response,                            // buffer to store response JSON
    sizeof(response),                    // size of response buffer
    options,                             // optional: generation options (nullptr for defaults)
    nullptr,                             // optional: tools JSON for function calling 
    nullptr,                             // optional: streaming callback fn(token, id, user_data)
    nullptr                              // optional: user data passed to callback
);

Why Use Cactus

Cactus stands out as a robust solution for on-device AI inference due to several key advantages:

Cross-Platform & Energy-Efficient: Designed from the ground up for mobile devices, ensuring optimal performance and low power consumption on both iOS and Android.
NPU Support: Leverages Neural Processing Units (NPUs) on compatible devices, such as Apple Silicon, for real-time inference and handling large contexts.
Mixed Precision: Intelligently blends INT4, INT8, and FP16 for weights, achieving optimal balance between model size, speed, and accuracy.
High Performance: Offers impressive decode and prefill tokens per second, as demonstrated across various devices from Mac M4 Pro to Raspberry Pi 5.
Extensive Model Support: Supports a wide range of popular models including Gemma, Whisper, LFM2, and Qwen, with capabilities for completion, tools, vision, embeddings, and speech.
Developer-Friendly Ecosystem: Provides SDKs for Python, React Native, Swift, Kotlin, Flutter, and Rust, making integration into existing applications seamless.