ds4: A Fast Local Inference Engine for DeepSeek V4 Flash and PRO on Metal, CUDA, ROCm

Introduction

ds4 is a specialized, native inference engine meticulously optimized for DeepSeek V4 Flash. It also extends support to GLM 5.2 and, for high-memory systems, DeepSeek V4 PRO. Unlike general GGUF runners, ds4 is self-contained and deliberately narrow, focusing on a curated set of models to achieve peak performance. It integrates model loading, prompt rendering, tool calls, KV state management, an HTTP server, and a coding agent, all built and tested cohesively. ds4 supports a range of powerful backends, including Apple's Metal for Macs, NVIDIA CUDA for various GPU systems, and ROCm for AMD Strix Halo systems. The project acknowledges its foundational debt to llama.cpp and GGML, which paved the way for its development.

Installation

To get started with ds4, you first need to download the desired model weights. The repository provides a convenient script for this:

./download_model.sh ds4f-q2 # Example for DeepSeek V4 Flash Q2

After downloading your model, you can build ds4 for your specific hardware backend. Here are the common build commands:

make                  # macOS Metal
make cuda-spark       # Linux CUDA, DGX Spark / GB10
make cuda-generic     # Linux CUDA, other local CUDA GPUs
make strix-halo       # Linux ROCm, AMD Strix Halo
make cpu              # CPU-only diagnostics build

Once built, you can run ds4 using the ./ds4 executable. By default, it uses ds4flash.gguf, but you can specify a different model with the -m flag, for example:

./ds4 -m gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf

Examples

ds4 offers various ways to interact with the models, from a simple command-line interface to advanced distributed setups.

CLI Interaction

You can run one-shot prompts or engage in an interactive chat session:

./ds4 -p "Explain Redis streams in one paragraph."

For an interactive multi-turn chat, simply run:

./ds4
ds4>

SSD Streaming for Large Models

For models exceeding your GPU's RAM, ds4 supports SSD streaming on Metal and ROCm. This allows you to run larger models by loading routed MoE experts from disk as needed:

./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB

Distributed Inference with Pipeline Parallelism

Split transformer layers across multiple machines to run models too large for a single host or to accelerate prefill. For example, to run DeepSeek V4 PRO Q4 across two Mac Studios:

On Machine A (coordinator):

./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf --role coordinator --layers 0:30 --listen 169.254.43.68 1234

On Machine B (worker):

./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf --role worker --layers 31:output --coordinator 169.254.43.68 1234

Tensor Parallelism

Achieve lower per-token latency by splitting the heavy per-layer work of a single decode across two Macs connected with Thunderbolt 5, or across multiple CUDA GPUs.

Example for two Macs (after sysctl and ifconfig setup):

On Machine B (worker):

./ds4 -m "MODEL_PATH.gguf" --tensor-parallel --role worker --coordinator 10.99.0.2 9911 --transport rdma

On Machine A (coordinator):

./ds4 -m "MODEL_PATH.gguf" --tensor-parallel --role coordinator --listen 10.99.0.2 9911 --transport rdma -c 8192 -p "Tell me something about the sea."

For CUDA multi-GPU tensor parallelism:

./ds4-agent --cuda --cuda-tensor-parallel --gpu-vram auto --gpu-devices 0,2,4,6,1,3,5,7 --model "MODEL_PATH.gguf" --ctx 100000

OpenAI/Anthropic Compatible Server

Host a local server for API access, supporting batched sessions and disk KV caching:

./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

Native Agent

ds4 includes a native coding agent with low latency, direct tool calling, and session management via on-disk KV cache. Start it with ./ds4-agent.

Why Use ds4

ds4 stands out as a powerful solution for local LLM inference due to several key advantages. It enables users to run highly capable open-weight models, such as DeepSeek V4 Flash and PRO, directly on consumer hardware like MacBooks, DGX Spark, or Strix Halo systems. The project leverages aggressive routed-expert quantization and compressed KV caches, making long contexts practical and efficient even with fast local SSDs. ds4 is designed as a specialized inference system, focusing on a few models to deliver unparalleled optimization and performance. This specialization allows for advanced features like multi-GPU LLM serving, pipeline parallelism to combine RAM from multiple systems, and tensor parallelism to reduce per-token latency. Furthermore, ds4 offers options to reduce power consumption and heat, making long inference runs more sustainable, and includes a native agent for a seamless, low-latency coding experience.

Links

• GitHub Repository: https://github.com/antirez/ds4
• Hugging Face Models (antirez): https://huggingface.co/antirez/deepseek-v4-gguf