Our take
Activeregular commits and releases- Activity
- 68
- Community
- 4
- Issues
- 43
- Pull requests
- 20
+35 stars in the last 30 days
- Fewer contributors than 91% of the projects we track
Shimmy is an actively maintained, unusually focused Rust inference server, but adopting it means relying on a project whose measured development and maintenance work is concentrated in one contributor.
Good fit if
- You want a local OpenAI-compatible API for GGUF models and are comfortable with the Airframe/WebGPU stack.
- Your deployment can use Rust and a supported GPU, and the model you need appears in the certified-model documentation.
- You value a permissive MIT-licensed project and can account for its single-maintainer concentration.
Look elsewhere if
- You need an inference backend other than Airframe, such as llama.cpp, which the README says was removed from Shimmy in v2.
- You require full native SafeTensors inference today, rather than the loading support and roadmap work described by the project.
- Your target environment cannot run the required Rust build or WebGPU GPU path.
All health signals
| Last commit | 2026-11-01 (1 day ago) |
|---|---|
| Commits, last 90 days | 74 |
| Releases, last 12 months | 17 (latest v2.6.5, 2026-10-19) |
| Contributors | 1 |
| Issues closed, last 90 days | 5 (typically closed in 38 days) |
| Pull requests merged, last 90 days | 0 |
| Project age | 14 months |
Checked on 2026-11-03 with the GitHub API.
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
6,014Forks
589Change since Nov 14, 2026: +1 stars·0 forks
Overview
Shimmy is a local inference server for GGUF models, built on Airframe, a pure-Rust WebGPU transformer engine. It exposes an OpenAI-compatible API so existing SDKs and tools can send chat and completion requests to models running on the user's machine, rather than a hosted service. The project describes support for NVIDIA, AMD, Intel, integrated GPUs, and Apple Silicon through WebGPU.
Shimmy is the server layer; Airframe is its inference engine. That distinction matters for adoption: this is a focused Rust and WebGPU stack, not a wrapper around llama.cpp or a general-purpose model platform. The README lists certified model and quantization combinations across several model families, and notes that SafeTensors loading is supported while full native SafeTensors inference remains roadmap work.
Key Features
- Serves local GGUF models through OpenAI-compatible chat completions, text completions, streaming, and model endpoints.
- Uses Airframe, a pure-Rust WebGPU engine with WGSL compute shaders, rather than Python or a C++ inference backend.
- Supports GGUF model metadata-driven configuration and lists certified model/quantization combinations in the project documentation.
- Offers an INT4 KV cache option, which the project says reduces cache memory in tested configurations.
- Provides YaRN RoPE scaling for extended context through
SHIMMY_MAX_CTX. - Requires an API key for network binds; loopback access does not require one, and browser clients need configured allowed origins.
Use Cases
- Run a compatible GGUF model on a workstation and connect existing OpenAI SDK-based applications to a local endpoint.
- Keep prompts and model requests on a local machine for development or workflows where sending data to a hosted inference API is undesirable.
- Build Rust-based tooling that needs an OpenAI-style chat or completion API without adding a Python runtime or C++ toolchain.
- Experiment with certified quantized models on a WebGPU-capable GPU, after checking the model and hardware requirements in the documentation.
What you need
Detected in the repository
- Rust (from Cargo.toml)
- A Dockerfile, so it can run in a container; a Compose file sets up the related services
- A test suite and automated checks on GitHub Actions
License in plain words
Apache-2.0permissive
- Commercial use: yes
- Modify and redistribute: yes
- You must keep: the license, the NOTICE file and a note of your changes
- Share your changes: no
- Includes an explicit patent grant from the contributors:
A summary, not legal advice: the LICENSE file is what applies.
Getting Started
Install and start the server with a model file:
cargo install shimmy --locked
shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435
See the quick-start guide for model setup, GPU guidance, and platform-specific instructions.
Alternatives
- OGAD: OGAD is a cross-platform local AI studio using llama.cpp and bundled runtimes, while Shimmy is a single-binary GGUF server built on Rust and WebGPU.
- SwarmLLM: SwarmLLM can pool computing resources across computers and offers Anthropic-compatible APIs; Shimmy serves GGUF models on supported local GPUs.
- LocalAI: LocalAI supports multiple model types and distributed hardware, while Shimmy focuses on GGUF inference through a Rust WebGPU engine.
- textgen: TextGen centers on a private desktop and web interface for managing models, while Shimmy is a lightweight OpenAI-compatible inference server.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| shimmy | Rust | Apache-2.0 | 6k | Active |
| OGAD | TypeScript | AGPL-3.0 | 135 | Active |
| SwarmLLM | Rust | Apache-2.0 | 55 | Active |
| LocalAI | Go | MIT | 49.4k | Active |
| textgen | Python | AGPL-3.0 | 47.7k | Active |
Considerations
- GPU inference depends on WebGPU support and available GPU memory; check the project's hardware and VRAM guidance before selecting a model.
- The project reports frequent recent commits and releases, but measured contributor data shows that all contribution activity is concentrated in one person. Teams should consider their tolerance for maintainer availability and continuity.
- The README distinguishes certified model/quantization combinations from architecture recognition. A model family being recognized does not guarantee that a particular file or quantization is certified.
- SafeTensors loading is available, but full Airframe-native SafeTensors inference is described as roadmap work.
- The repository metadata lists Apache-2.0, while the README states MIT. Confirm the applicable license in the repository's current license file before incorporating or redistributing the software.
Found this useful?
Share it with someone who would like shimmy.