# Cactus: Cross-Platform AI Inference Engine for Mobile Devices

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/cactus-compute-cactus
Generated for open source discovery and AI-assisted research.

Cactus is an open-source project providing an energy-efficient, cross-platform AI inference engine specifically designed for mobile devices. It features low-level ARM-specific SIMD operations, a unified zero-copy computation graph, and a high-level transformer engine with NPU support. This framework enables developers to deploy state-of-the-art AI models on smartphones and other edge devices with impressive performance.

GitHub: https://github.com/cactus-compute/cactus
OSRepos URL: https://osrepos.com/repo/cactus-compute-cactus

## Summary

Cactus is an open-source project providing an energy-efficient, cross-platform AI inference engine specifically designed for mobile devices. It features low-level ARM-specific SIMD operations, a unified zero-copy computation graph, and a high-level transformer engine with NPU support. This framework enables developers to deploy state-of-the-art AI models on smartphones and other edge devices with impressive performance.

## Topics

- ai
- android
- edge
- ios
- llm-inference
- mobile
- C++
- framework

## Repository Information

Last analyzed by OSRepos: Sun Jan 25 2026 20:01:08 GMT+0000 (Western European Standard Time)
Detail views: 15
GitHub clicks: 16

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

Cactus is a powerful open-source project by Cactus Compute, Inc. and UCLA's BruinAI, offering an advanced AI inference engine optimized for mobile devices. It provides a comprehensive stack, from low-level hardware-specific kernels to a high-level API, enabling efficient execution of AI models on various platforms like iOS and Android.

The architecture of Cactus is layered, starting with `Cactus Kernels` for ARM-specific SIMD operations, building up to `Cactus Graph` for unified computation, `Cactus Models` for implementing SOTA models, `Cactus Engine` for transformer inference with NPU support, and finally `Cactus FFI` for an OpenAI-compatible C API.

## Installation

Getting started with Cactus on a Mac is straightforward. First, clone the repository and navigate into its directory. Then, source the setup script to configure your environment.

bash
git clone https://github.com/cactus-compute/cactus && cd cactus && source ./setup


Remember to run `source ./setup` in any new terminal session. You can use `cactus run [model]` to open a playground, `cactus download [model]` to get models, and `cactus build` to compile for ARM targets like Apple or Android.

## Examples

Cactus provides intuitive APIs for both graph-level operations and high-level model inference. Below are examples demonstrating how to use `Cactus Graph & Kernel` for custom computations and `Cactus Engine & FFI` for running LLM inference.

### Cactus Graph & Kernel

This example shows how to define a computation graph, set inputs, execute it, and retrieve results using Cactus's low-level graph API.

cpp
#include cactus.h

CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);

auto x1 = graph.matmul(a, b, false);
auto x2 = graph.transpose(x1);
auto result = graph.matmul(b, x2, true);

float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f};
float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};

graph.set_input(a, a_data, Precision::FP16);
graph.set_input(b, b_data, Precision::INT8);

graph.execute();
void* output_data = graph.get_output(result);

graph.hard_reset(); 



### Cactus Engine & FFI

This snippet demonstrates using the `Cactus Engine` via its FFI to perform AI inference, similar to OpenAI's API, with support for chat messages and generation options.

cpp
#include cactus.h

cactus_set_pro_key("");                  // email founders@cactuscompute.com for optional key

cactus_model_t model = cactus_init(
    "path/to/weight/folder",             // section to generate weigths below
    "txt/or/md/file/or/dir/with/many",   // nullptr if none, cactus does automatic fast RAG
);

const char* messages = R"([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Henry Ndubuaku"}
])";

const char* options = R"({
    "max_tokens": 50,
    "stop_sequences": ["<|im_end|>"]
})";

char response[4096];
int result = cactus_complete(
    model,                               // model handle from cactus_init
    messages,                            // JSON array of chat messages
    response,                            // buffer to store response JSON
    sizeof(response),                    // size of response buffer
    options,                             // optional: generation options (nullptr for defaults)
    nullptr,                             // optional: tools JSON for function calling 
    nullptr,                             // optional: streaming callback fn(token, id, user_data)
    nullptr                              // optional: user data passed to callback
);


## Why Use Cactus

Cactus stands out as a robust solution for on-device AI inference due to several key advantages:

*   **Cross-Platform & Energy-Efficient**: Designed from the ground up for mobile devices, ensuring optimal performance and low power consumption on both iOS and Android.
*   **NPU Support**: Leverages Neural Processing Units (NPUs) on compatible devices, such as Apple Silicon, for real-time inference and handling large contexts.
*   **Mixed Precision**: Intelligently blends INT4, INT8, and FP16 for weights, achieving optimal balance between model size, speed, and accuracy.
*   **High Performance**: Offers impressive decode and prefill tokens per second, as demonstrated across various devices from Mac M4 Pro to Raspberry Pi 5.
*   **Extensive Model Support**: Supports a wide range of popular models including Gemma, Whisper, LFM2, and Qwen, with capabilities for completion, tools, vision, embeddings, and speech.
*   **Developer-Friendly Ecosystem**: Provides SDKs for Python, React Native, Swift, Kotlin, Flutter, and Rust, making integration into existing applications seamless.

## Links

Explore Cactus further through its official repository and various SDKs and demo applications:

*   **GitHub Repository**: [https://github.com/cactus-compute/cactus](https://github.com/cactus-compute/cactus){:target="_blank"}
*   **Python SDK for Mac**: [https://github.com/cactus-compute/cactus/tree/main/python](https://github.com/cactus-compute/cactus/tree/main/python){:target="_blank"}
*   **React Native SDK**: [https://github.com/cactus-compute/cactus-react-native](https://github.com/cactus-compute/cactus-react-native){:target="_blank"}
*   **Swift Multiplatform SDK**: [https://github.com/mhayes853/swift-cactus](https://github.com/mhayes853/swift-cactus){:target="_blank"}
*   **Kotlin Multiplatform SDK**: [https://github.com/cactus-compute/cactus-kotlin](https://github.com/cactus-compute/cactus-kotlin){:target="_blank"}
*   **Flutter SDK**: [https://github.com/cactus-compute/cactus-flutter](https://github.com/cactus-compute/cactus-flutter){:target="_blank"}
*   **Rust SDK**: [https://github.com/mrsarac/cactus-rs](https://github.com/mrsarac/cactus-rs){:target="_blank"}
*   **iOS Demo App**: [https://apps.apple.com/gb/app/cactus-chat/id6744444212](https://apps.apple.com/gb/app/cactus-chat/id6744444212){:target="_blank"}
*   **Android Demo App**: [https://play.google.com/store/apps/details?id=com.rshemetsubuser.myapp](https://play.google.com/store/apps/details?id=com.rshemetsubuser.myapp){:target="_blank"}