ds4: Run DeepSeek and Other LLMs Locally

Overview

ds4 is a native inference engine, called DwarfStar in its README, focused on running a selected set of open-weight language models on local hardware. Rather than acting as a general GGUF runner, it targets specific model families and the GGUF files produced for this project.

It is intended for people who want to run models on high-memory Macs, CUDA systems, or supported ROCm machines, including setups that use SSD streaming or multiple machines. The project also bundles command-line, server, and coding-agent workflows around its inference engine.

Key Features

• Targets DeepSeek V4 Flash and PRO, GLM, and Qwen models, with support varying by model and backend.
• Supports Apple Metal, NVIDIA CUDA, and ROCm on supported hardware.
• Provides an interactive CLI, a local HTTP server, and a native coding agent.
• Offers SSD streaming for some setups that cannot keep the full model in memory.
• Includes multi-GPU support and distributed execution options such as tensor and pipeline parallelism.
• Supports vision for selected models, plus optional speculative decoding for supported configurations.
• Includes integration evaluation tools and model-building utilities.

Use Cases

• Run selected open-weight models locally on a high-memory Mac, CUDA system, or supported ROCm machine.
• Serve local model inference through an HTTP API for compatible clients or multiple sessions.
• Use the native coding agent for programming tasks without running a separate inference server.
• Experiment with larger models using SSD streaming or multi-machine setups when the hardware and model are supported.

Getting Started

git clone https://github.com/antirez/ds4.git
cd ds4

Choose a platform-specific build and model from the README and its linked guides.

Alternatives

• colibri: colibri focuses on streaming Mixture-of-Experts weights from disk, while ds4 targets selected models with native Metal, CUDA, and ROCm inference.
• TurboLLM: TurboLLM offers a browser UI and compatible APIs across multiple inference engines, while ds4 is a native inference engine for selected models.

Considerations

• This is a model-specific engine, not a general-purpose GGUF runner. Use the GGUF files produced for the project.
• Hardware, model, and feature support varies. The README documents memory requirements and separate build paths; some use cases need substantial RAM, a fast SSD, or multiple GPUs or machines.
• The README describes the software as beta quality and warns that instability and regressions are possible.
• Model support is explicitly opportunistic and may change as newer models become available.