# ds4: A Fast Local Inference Engine for DeepSeek V4 Flash and PRO on Metal, CUDA, ROCm

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/antirez-ds4
Generated for open source discovery and AI-assisted research.

ds4 is a highly optimized, native inference engine designed for DeepSeek V4 Flash and PRO models. It provides efficient local inference across various hardware platforms, including Apple Silicon (Metal), NVIDIA GPUs (CUDA), and AMD ROCm. This project focuses on delivering high performance for large language models on consumer-grade machines.

GitHub: https://github.com/antirez/ds4
OSRepos URL: https://osrepos.com/repo/antirez-ds4

## Summary

ds4 is a highly optimized, native inference engine designed for DeepSeek V4 Flash and PRO models. It provides efficient local inference across various hardware platforms, including Apple Silicon (Metal), NVIDIA GPUs (CUDA), and AMD ROCm. This project focuses on delivering high performance for large language models on consumer-grade machines.

## Topics

- C
- AI
- LLM
- Inference Engine
- Deep Learning
- Metal
- CUDA
- ROCm

## Repository Information

Last analyzed by OSRepos: Wed Aug 26 2026 08:23:13 GMT+0100 (Western European Summer Time)
Detail views: 0
GitHub clicks: 0

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction
`ds4` is a specialized, native inference engine meticulously optimized for DeepSeek V4 Flash. It also extends support to GLM 5.2 and, for high-memory systems, DeepSeek V4 PRO. Unlike general GGUF runners, `ds4` is self-contained and deliberately narrow, focusing on a curated set of models to achieve peak performance. It integrates model loading, prompt rendering, tool calls, KV state management, an HTTP server, and a coding agent, all built and tested cohesively. `ds4` supports a range of powerful backends, including Apple's Metal for Macs, NVIDIA CUDA for various GPU systems, and ROCm for AMD Strix Halo systems. The project acknowledges its foundational debt to `llama.cpp` and GGML, which paved the way for its development.

## Installation
To get started with `ds4`, you first need to download the desired model weights. The repository provides a convenient script for this:
sh
./download_model.sh ds4f-q2 # Example for DeepSeek V4 Flash Q2

After downloading your model, you can build `ds4` for your specific hardware backend. Here are the common build commands:
sh
make                  # macOS Metal
make cuda-spark       # Linux CUDA, DGX Spark / GB10
make cuda-generic     # Linux CUDA, other local CUDA GPUs
make strix-halo       # Linux ROCm, AMD Strix Halo
make cpu              # CPU-only diagnostics build

Once built, you can run `ds4` using the `./ds4` executable. By default, it uses `ds4flash.gguf`, but you can specify a different model with the `-m` flag, for example:
sh
./ds4 -m gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf


## Examples
`ds4` offers various ways to interact with the models, from a simple command-line interface to advanced distributed setups.

### CLI Interaction
You can run one-shot prompts or engage in an interactive chat session:
sh
./ds4 -p "Explain Redis streams in one paragraph."

For an interactive multi-turn chat, simply run:
sh
./ds4
ds4>


### SSD Streaming for Large Models
For models exceeding your GPU's RAM, `ds4` supports SSD streaming on Metal and ROCm. This allows you to run larger models by loading routed MoE experts from disk as needed:
sh
./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB


### Distributed Inference with Pipeline Parallelism
Split transformer layers across multiple machines to run models too large for a single host or to accelerate prefill. For example, to run DeepSeek V4 PRO Q4 across two Mac Studios:

On Machine A (coordinator):
sh
./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf --role coordinator --layers 0:30 --listen 169.254.43.68 1234

On Machine B (worker):
sh
./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf --role worker --layers 31:output --coordinator 169.254.43.68 1234


### Tensor Parallelism
Achieve lower per-token latency by splitting the heavy per-layer work of a single decode across two Macs connected with Thunderbolt 5, or across multiple CUDA GPUs.

Example for two Macs (after `sysctl` and `ifconfig` setup):
On Machine B (worker):
sh
./ds4 -m "MODEL_PATH.gguf" --tensor-parallel --role worker --coordinator 10.99.0.2 9911 --transport rdma

On Machine A (coordinator):
sh
./ds4 -m "MODEL_PATH.gguf" --tensor-parallel --role coordinator --listen 10.99.0.2 9911 --transport rdma -c 8192 -p "Tell me something about the sea."

For CUDA multi-GPU tensor parallelism:
sh
./ds4-agent --cuda --cuda-tensor-parallel --gpu-vram auto --gpu-devices 0,2,4,6,1,3,5,7 --model "MODEL_PATH.gguf" --ctx 100000


### OpenAI/Anthropic Compatible Server
Host a local server for API access, supporting batched sessions and disk KV caching:
sh
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192


### Native Agent
`ds4` includes a native coding agent with low latency, direct tool calling, and session management via on-disk KV cache. Start it with `./ds4-agent`.

## Why Use ds4
`ds4` stands out as a powerful solution for local LLM inference due to several key advantages. It enables users to run highly capable open-weight models, such as DeepSeek V4 Flash and PRO, directly on consumer hardware like MacBooks, DGX Spark, or Strix Halo systems. The project leverages aggressive routed-expert quantization and compressed KV caches, making long contexts practical and efficient even with fast local SSDs. `ds4` is designed as a specialized inference system, focusing on a few models to deliver unparalleled optimization and performance. This specialization allows for advanced features like multi-GPU LLM serving, pipeline parallelism to combine RAM from multiple systems, and tensor parallelism to reduce per-token latency. Furthermore, `ds4` offers options to reduce power consumption and heat, making long inference runs more sustainable, and includes a native agent for a seamless, low-latency coding experience.

## Links
*   GitHub Repository: [https://github.com/antirez/ds4](https://github.com/antirez/ds4){:target="_blank"}
*   Hugging Face Models (antirez): [https://huggingface.co/antirez/deepseek-v4-gguf](https://huggingface.co/antirez/deepseek-v4-gguf){:target="_blank"}