SwarmLLM: Run Local AI Models and Team Up for Giant Distributed Inference

Introduction

SwarmLLM is an innovative, open-source application designed to bring the power of AI chat models to your local machine. Developed in Rust, it provides a ChatGPT-style assistant using open AI models from various providers like Meta, Google, and Mistral. What sets SwarmLLM apart is its ability to facilitate distributed inference, allowing multiple computers to collaborate and run AI models that would otherwise be too large for any single machine. This means you can access more powerful AI, leveraging a collective network of resources.

Why Use SwarmLLM and Key Benefits

SwarmLLM offers a compelling alternative to cloud-based AI services with several significant advantages:

• Local AI Chat: Run AI models directly on your computer. Once downloaded, your conversations remain private, as there's no external company involved.
• Distributed Inference: Access models too large for your machine. Computers in the swarm each hold a part of a larger model, working together to answer your questions. This is fastest when computers are geographically close.
• Community Contribution: By leaving SwarmLLM running, your computer can help the swarm by sharing model parts and assisting with other users' requests, similar to seeding a torrent. In return, you gain access to the collective power of the swarm.
• Device Linking: Create a private group with your own devices, such as a laptop, desktop, and home server, to utilize them as one larger machine.
• API Compatibility: Integrate SwarmLLM with existing applications, coding assistants, and AI agents that are compatible with ChatGPT or Claude, using a local OpenAI- and Anthropic-compatible endpoint.
• No Hidden Costs: The project is free, open-source, and operates without accounts, subscriptions, ads, or cryptocurrency. The only cost is your electricity and internet usage while contributing.
• Privacy Options: While models running entirely on your machine ensure privacy, the public swarm offers partial privacy with encrypted connections. SwarmLLM also provides 'Private Mode' to keep questions within your linked devices or local network.

Installation

Getting started with SwarmLLM is straightforward. You can find the latest releases and detailed instructions on the SwarmLLM GitHub releases page.

For Windows:

1. Download swarmllm-windows-x86_64-gpu.zip if you have an NVIDIA, AMD, or Intel graphics card. Otherwise, use swarmllm-windows-x86_64-cpu.zip.
2. Extract the downloaded ZIP file.
3. Double-click swarmllm.exe. You may need to click "More info" then "Run anyway" if Windows displays a security warning.
4. Leave the black console window open, as it signifies SwarmLLM is running. The app will open in your web browser automatically.

For Mac (Apple Silicon M1 or newer):

1. Download swarmllm-macos-aarch64.tar.gz and double-click to unpack.
2. Keep the folder in Documents or Downloads.
3. Double-click swarmllm. If macOS blocks the launch, go to System Settings ? Privacy & Security, click "Open Anyway", and double-click again.
4. A Terminal window will open, indicating SwarmLLM is running. The app will appear in your web browser.

For Linux:

1. Download swarmllm-linux-x86_64-cuda.tar.gz for NVIDIA RTX 30-series or newer, or swarmllm-linux-x86_64.tar.gz otherwise.
2. Unpack the archive and run ./swarmllm run.

After installation, pick a model within the app and start chatting. The first model download is a one-time process of a few gigabytes. If the browser doesn't open, navigate to localhost:8800.

Examples

SwarmLLM provides an OpenAI- and Anthropic-compatible endpoint, allowing developers to integrate it into their existing tools. Here are examples for using the API:

OpenAI-compatible API call:

export SWARMLLM_KEY=...   # from Settings ? Access Token

curl http://localhost:8800/v1/chat/completions \
  -H "Authorization: Bearer $SWARMLLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3.2-3b-instruct-q4-k-m",
       "messages": [{"role": "user", "content": "Hello!"}]}'

As a Claude Code backend:

ANTHROPIC_BASE_URL=http://localhost:8800 ANTHROPIC_AUTH_TOKEN="$SWARMLLM_KEY" \
  claude --model qwen2.5-coder-7b-instruct-q4-k-m

Links

• GitHub Repository: https://github.com/enapt/SwarmLLM
• Latest Releases: https://github.com/enapt/SwarmLLM/releases/latest
• Discord Community: https://discord.gg/nq9be3u828
• Installation Guide: https://enapt.github.io/SwarmLLM/getting-started/installation.html
• API Reference: https://enapt.github.io/SwarmLLM/api/openai.html
• Issues: https://github.com/enapt/SwarmLLM/issues
• Contributing: https://github.com/enapt/SwarmLLM/blob/main/CONTRIBUTING.md