SwarmLLM: Run Local and Distributed AI Models

Overview

SwarmLLM is a Rust application for chatting with open language models locally or using a peer-to-peer network of computers to run models that exceed one machine’s capacity. It addresses the gap between single-computer local inference and larger models that usually require more hardware.

It is aimed at people who want self-hosted AI, as well as developers who need a local-compatible endpoint. Distributed use comes with a privacy tradeoff: participating computers can see the work they process, so the project advises against sending sensitive information through the public swarm.

Key Features

• Run supported open models locally, with the app indicating which models fit the computer.
• Distribute model parts across peer computers and coordinate inference over the network.
• Link personal devices into a private group for shared inference.
• Expose OpenAI- and Anthropic-compatible APIs at http://localhost:8800.
• Support streaming, local-model tool calling, an MCP server, and optional routing to cloud providers using the user’s own keys.
• Provide controls for storage, contribution levels, and resource schedules.
• Verify model parts when they arrive and load.

Use Cases

• Individuals can run supported models on their own computer when they want a self-hosted alternative to cloud chat services.
• Developers can connect compatible chat applications or coding agents to local models through the API.
• People with several computers can pool their devices to run models that do not fit on one machine.
• Community members can contribute spare computing resources to help others run models through the swarm.

Getting Started

Download the package for your operating system from the latest releases, then follow the platform-specific steps in the README. To build from source, the README gives this command:

git clone https://github.com/enapt/SwarmLLM.git && cd SwarmLLM
cargo build --release

Alternatives

• LocalAI: LocalAI serves a broad range of model types through compatible APIs, while SwarmLLM focuses on pooling computers to run larger language models.
• jan: Jan centers on a desktop assistant and local model management, while SwarmLLM centers on distributed inference and APIs for apps and agents.
• textgen: TextGen emphasizes a web interface for managing and using local models, while SwarmLLM emphasizes sharing compute across computers.

Considerations

• The README describes the software as an early version that is still improving.
• Building from source requires Rust 1.90 or later.
• Model downloads can take several gigabytes, and the app has a configurable disk limit.
• A GPU can improve performance, but support varies by operating system; the README notes that Mac support is for Apple-chip models and Linux GPU packages target NVIDIA RTX 30-series or newer.
• Distributed inference depends on available peers and network proximity. Computers in the public swarm can see the work they process, so it is not suitable for sensitive prompts.
• The README reports substantial idle network traffic in its measurements and cautions that the application may use too much data for capped connections.
• The README advertises dual MIT and Apache 2.0 licensing, while the supplied repository metadata lists Apache-2.0.