cactus: Run AI Inference on Phones and Wearables

Overview

Cactus is a hybrid edge-cloud AI engine for mobile devices and wearables. It combines a model runtime, computation graph, device-oriented kernels, and custom quantization to support local inference for text, vision, and speech.

It is aimed at developers building AI features for phones, tablets, wearables, smart-home devices, and robots. Its local-first approach can reduce dependence on network access, while optional cloud handoff provides a route for queries a local model cannot handle confidently.

Key Features

• Runs language, vision, and speech inference through a shared engine.
• Provides a C API for chat completion, streaming, tool calling, transcription, embeddings, retrieval-augmented generation, and related engine functions.
• Includes a C++ computation graph and CPU/GPU kernels, including ARM NEON kernels.
• Converts model weights with Cactus Quants, using rotation-based quantization options from 1-bit to 4-bit, including mixed-precision modes.
• Offers an OpenAI-compatible local HTTP server and command-line tools for running, converting, downloading, and benchmarking models.
• Supports optional cloud handoff based on local model confidence.
• Provides bindings for Swift, Kotlin, Flutter, React Native, Python, and Rust.

Use Cases

• Mobile app developers can add on-device chat or vision features where network availability or response latency matters.
• Wearable and embedded-device teams can evaluate compact quantized models for constrained hardware.
• Speech application developers can run transcription on supported devices without routing every request to a remote service.
• Teams building assistants can use local tool calling and retrieval, with cloud handoff available for difficult queries.

Project Facts

• Language: C++
• License: NOASSERTION
• Stars: 6.1k
• Forks: 512
• Topics: ai, android, arm, edge, edge-ai, framework, ios, llamacpp, llm, llm-inference, llms, mobile, mobile-inference, on-device-ai, quantiz, rag, smartphone, speech, transformer, whisper
• Archived: false

Getting Started

On a Mac, install and run the CLI:

brew install cactus-compute/cactus/cactus
cactus run

See the repository README for build instructions, model commands, platform options, and API details.

Alternatives

• whisper.cpp: whisper.cpp focuses on offline speech recognition, while Cactus supports language, vision, and speech inference on mobile and edge devices.
• colibri: colibri focuses on streaming Mixture-of-Experts weights from disk, while Cactus targets broader model types and mobile and edge deployment.

Considerations

• Hardware-specific performance varies, and the README's benchmark results cover only the listed devices and models.
• The README describes conversion for arbitrary Hugging Face models as experimental. Liquid, Gemma, Whisper, Parakeet, and Qwen families are identified as especially tested.
• Building from source on Linux requires Python, CMake, build tools, and libcurl development headers, according to the README.
• The repository reports its license as NOASSERTION, so verify licensing terms before adopting it in a project.
• Cloud handoff is optional, but using it may require cloud credentials and introduces a network dependency for those requests.