{"name":"ds4: A Fast Local Inference Engine for DeepSeek V4 Flash and PRO on Metal, CUDA, ROCm","description":"ds4 is a highly optimized, native inference engine designed for DeepSeek V4 Flash and PRO models. It provides efficient local inference across various hardware platforms, including Apple Silicon (Metal), NVIDIA GPUs (CUDA), and AMD ROCm. This project focuses on delivering high performance for large language models on consumer-grade machines.","github":"https://github.com/antirez/ds4","url":"https://osrepos.com/repo/antirez-ds4","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/antirez-ds4","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/antirez-ds4.md","json":"https://osrepos.com/repo/antirez-ds4.json","topics":["C","AI","LLM","Inference Engine","Deep Learning","Metal","CUDA","ROCm"],"keywords":["C","AI","LLM","Inference Engine","Deep Learning","Metal","CUDA","ROCm"],"stars":null,"summary":"ds4 is a highly optimized, native inference engine designed for DeepSeek V4 Flash and PRO models. It provides efficient local inference across various hardware platforms, including Apple Silicon (Metal), NVIDIA GPUs (CUDA), and AMD ROCm. This project focuses on delivering high performance for large language models on consumer-grade machines.","content":"## Introduction\n`ds4` is a specialized, native inference engine meticulously optimized for DeepSeek V4 Flash. It also extends support to GLM 5.2 and, for high-memory systems, DeepSeek V4 PRO. Unlike general GGUF runners, `ds4` is self-contained and deliberately narrow, focusing on a curated set of models to achieve peak performance. It integrates model loading, prompt rendering, tool calls, KV state management, an HTTP server, and a coding agent, all built and tested cohesively. `ds4` supports a range of powerful backends, including Apple's Metal for Macs, NVIDIA CUDA for various GPU systems, and ROCm for AMD Strix Halo systems. The project acknowledges its foundational debt to `llama.cpp` and GGML, which paved the way for its development.\n\n## Installation\nTo get started with `ds4`, you first need to download the desired model weights. The repository provides a convenient script for this:\nsh\n./download_model.sh ds4f-q2 # Example for DeepSeek V4 Flash Q2\n\nAfter downloading your model, you can build `ds4` for your specific hardware backend. Here are the common build commands:\nsh\nmake                  # macOS Metal\nmake cuda-spark       # Linux CUDA, DGX Spark / GB10\nmake cuda-generic     # Linux CUDA, other local CUDA GPUs\nmake strix-halo       # Linux ROCm, AMD Strix Halo\nmake cpu              # CPU-only diagnostics build\n\nOnce built, you can run `ds4` using the `./ds4` executable. By default, it uses `ds4flash.gguf`, but you can specify a different model with the `-m` flag, for example:\nsh\n./ds4 -m gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf\n\n\n## Examples\n`ds4` offers various ways to interact with the models, from a simple command-line interface to advanced distributed setups.\n\n### CLI Interaction\nYou can run one-shot prompts or engage in an interactive chat session:\nsh\n./ds4 -p \"Explain Redis streams in one paragraph.\"\n\nFor an interactive multi-turn chat, simply run:\nsh\n./ds4\nds4>\n\n\n### SSD Streaming for Large Models\nFor models exceeding your GPU's RAM, `ds4` supports SSD streaming on Metal and ROCm. This allows you to run larger models by loading routed MoE experts from disk as needed:\nsh\n./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB\n\n\n### Distributed Inference with Pipeline Parallelism\nSplit transformer layers across multiple machines to run models too large for a single host or to accelerate prefill. For example, to run DeepSeek V4 PRO Q4 across two Mac Studios:\n\nOn Machine A (coordinator):\nsh\n./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf --role coordinator --layers 0:30 --listen 169.254.43.68 1234\n\nOn Machine B (worker):\nsh\n./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf --role worker --layers 31:output --coordinator 169.254.43.68 1234\n\n\n### Tensor Parallelism\nAchieve lower per-token latency by splitting the heavy per-layer work of a single decode across two Macs connected with Thunderbolt 5, or across multiple CUDA GPUs.\n\nExample for two Macs (after `sysctl` and `ifconfig` setup):\nOn Machine B (worker):\nsh\n./ds4 -m \"MODEL_PATH.gguf\" --tensor-parallel --role worker --coordinator 10.99.0.2 9911 --transport rdma\n\nOn Machine A (coordinator):\nsh\n./ds4 -m \"MODEL_PATH.gguf\" --tensor-parallel --role coordinator --listen 10.99.0.2 9911 --transport rdma -c 8192 -p \"Tell me something about the sea.\"\n\nFor CUDA multi-GPU tensor parallelism:\nsh\n./ds4-agent --cuda --cuda-tensor-parallel --gpu-vram auto --gpu-devices 0,2,4,6,1,3,5,7 --model \"MODEL_PATH.gguf\" --ctx 100000\n\n\n### OpenAI/Anthropic Compatible Server\nHost a local server for API access, supporting batched sessions and disk KV caching:\nsh\n./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192\n\n\n### Native Agent\n`ds4` includes a native coding agent with low latency, direct tool calling, and session management via on-disk KV cache. Start it with `./ds4-agent`.\n\n## Why Use ds4\n`ds4` stands out as a powerful solution for local LLM inference due to several key advantages. It enables users to run highly capable open-weight models, such as DeepSeek V4 Flash and PRO, directly on consumer hardware like MacBooks, DGX Spark, or Strix Halo systems. The project leverages aggressive routed-expert quantization and compressed KV caches, making long contexts practical and efficient even with fast local SSDs. `ds4` is designed as a specialized inference system, focusing on a few models to deliver unparalleled optimization and performance. This specialization allows for advanced features like multi-GPU LLM serving, pipeline parallelism to combine RAM from multiple systems, and tensor parallelism to reduce per-token latency. Furthermore, `ds4` offers options to reduce power consumption and heat, making long inference runs more sustainable, and includes a native agent for a seamless, low-latency coding experience.\n\n## Links\n*   GitHub Repository: [https://github.com/antirez/ds4](https://github.com/antirez/ds4){:target=\"_blank\"}\n*   Hugging Face Models (antirez): [https://huggingface.co/antirez/deepseek-v4-gguf](https://huggingface.co/antirez/deepseek-v4-gguf){:target=\"_blank\"}","metrics":{"detailViews":0,"githubClicks":0},"dates":{"published":null,"modified":"2026-08-26T07:23:13.000Z"}}