shimmy: Serve Local GGUF Models with an OpenAI-Compatible API

Overview

Shimmy is a single-binary server for running GGUF language models locally behind an OpenAI-compatible API. It aims to let existing SDKs and tools use local inference with minimal integration changes, while avoiding a Python runtime and C++ toolchain.

The project uses the separate Airframe Rust WebGPU inference engine. Shimmy is a fit for developers who want a self-hosted inference endpoint and whose hardware, model, and quantization are supported. It is not a general-purpose hosted model service, and compatibility with the OpenAI API does not mean every API feature or model is supported.

Key Features

• Serves local GGUF models through an OpenAI-compatible API, including chat completions, text completions, streaming, and model endpoints.
• Uses a Rust server and the Airframe WebGPU engine, without requiring Python or a C++ toolchain.
• Supports multiple model families and quantizations; the README distinguishes certified model-and-quant combinations from architecture recognition.
• Uses WebGPU compute shaders, with GPU support dependent on the available hardware and runtime.
• Offers INT4 KV-cache compression through TurboShimmy and YaRN RoPE scaling for extended context.
• Provides command-line model listing and server startup.

Use Cases

• Developers can point an OpenAI SDK or compatible tool at a local server instead of a remote inference endpoint.
• Teams can run supported GGUF models on their own machines when keeping prompts and inference local is important.
• Rust developers can evaluate a Rust-based inference serving setup without adding Python or a C++ build toolchain.
• Users can test supported models and quantizations through a local API before integrating them into an application.

Getting Started

Install and start the server with a local model file:

cargo install shimmy
shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435

See the README and its quickstart guide for model setup, GPU details, and configuration.

Alternatives

• llama-cpp-python: It wraps llama.cpp in Python and offers multiple APIs, while shimmy is a Rust server focused on GGUF inference.
• textgen: It centers on a desktop and web interface with broader model and feature support, while shimmy focuses on a lightweight OpenAI-compatible server.
• TurboLLM: It offers a browser interface and multiple inference engines, while shimmy is a Rust server for running GGUF models.

Considerations

• Inference depends on WebGPU support and available GPU resources; model size and quantization affect memory requirements.
• The README lists certified model-and-quant combinations separately from models that are merely recognized or supported. Check the supported models documentation before choosing a model.
• OpenAI-compatible describes the API surface, not complete compatibility with every OpenAI endpoint or client feature.
• The inference engine is maintained separately as Airframe, so engine capabilities and requirements are relevant when assessing Shimmy.