{"name":"StringWars: Benchmarking High-Performance String Processing in Rust and Python","description":"StringWars is a comprehensive GitHub repository dedicated to benchmarking performance-oriented string processing libraries in Rust and Python. It meticulously compares various operations, including substring search, hashing, and edit distances, across both CPUs and GPUs. This project serves as an invaluable resource for developers seeking to identify the fastest and most efficient solutions for critical string manipulation tasks, particularly those leveraging modern SIMD instructions and GPU acceleration.","github":"https://github.com/ashvardanian/StringWars","url":"https://osrepos.com/repo/ashvardanian-stringwars","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/ashvardanian-stringwars","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/ashvardanian-stringwars.md","json":"https://osrepos.com/repo/ashvardanian-stringwars.json","topics":["benchmark","string-processing","Rust","Python","performance","SIMD","GPU","bioinformatics"],"keywords":["benchmark","string-processing","Rust","Python","performance","SIMD","GPU","bioinformatics"],"stars":null,"summary":"StringWars is a comprehensive GitHub repository dedicated to benchmarking performance-oriented string processing libraries in Rust and Python. It meticulously compares various operations, including substring search, hashing, and edit distances, across both CPUs and GPUs. This project serves as an invaluable resource for developers seeking to identify the fastest and most efficient solutions for critical string manipulation tasks, particularly those leveraging modern SIMD instructions and GPU acceleration.","content":"## Introduction\n\nStringWars is an ambitious GitHub repository that rigorously benchmarks performance-oriented string processing libraries written in Rust and Python. The project aims to provide a clear comparison of various string operations, such as substring search, multi-pattern matching, hashing, edit distances, sketching, and sorting, across different hardware architectures, including CPUs and GPUs.\n\nDeveloped to accelerate the creation of the `StringZilla` C, C++, and CUDA libraries, StringWars highlights the simplicity of dependency management in Rust for benchmarking \"Systems Software.\" It focuses on workloads where modern hardware with wide SIMD instructions (like AVX-512 on x86 or SVE/SVE2 on Arm) can offer significant performance gains, often surpassing existing library capabilities. The benchmarks include comparisons against popular libraries like `memchr`, `rapidfuzz`, `aHash`, `aho_corasick`, `polars`, and `icu`.\n\n## Installation\n\nTo replicate the benchmark results, StringWars provides clear instructions for both Rust and Python environments.\n\n### Replicating Results in Rust\n\nFirst, clone the repository (a shallow clone is sufficient for benchmarks):\n\nbash\ngit clone --depth 1 https://github.com/ashvardanian/StringWars && cd StringWars\n\n\nThen, compile all dependencies with native CPU optimizations:\n\nbash\nRUSTFLAGS=\"-C target-cpu=native\" cargo build --benches --all-features\n\n\nTo run a specific benchmark, for example, the hash benchmark:\n\nbash\nRUSTFLAGS=\"-C target-cpu=native\" \\\n    STRINGWARS_DATASET=README.md \\\n    STRINGWARS_TOKENS=lines \\\n    cargo bench --features bench_hash --bench bench_hash --jobs $(nproc)\n\n\nFor GPU benchmarks, enable CUDA features explicitly:\n\nbash\nRUSTFLAGS=\"-C target-cpu=native\" \\\n    STRINGWARS_DATASET=README.md \\\n    STRINGWARS_TOKENS=lines \\\n    STRINGWARS_FILTER=GPU \\\n    cargo bench --features \"cuda bench_similarities\" --bench bench_similarities --jobs 1\n\n\n### Replicating Results in Python\n\nIt is recommended to use `uv` for Python dependency management. Install `uv` and then set up the environment:\n\nbash\nuv venv --python 3.12\nuv pip install -r requirements.txt -r requirements-cuda.txt\n\n\nTo run individual Python benchmarks, for example, the hash benchmark:\n\nbash\nuv run --no-project python hash/bench.py --help\n\n\nYou can also run Python benchmarks directly from a URL without cloning:\n\nbash\nuv run https://raw.githubusercontent.com/ashvardanian/StringWars/main/tokenization/bench.py \\\n    --dataset README.md --tokens file\n\n\n## Examples\n\nStringWars provides extensive benchmarks across various string operations. Here are a few highlights demonstrating the performance differences:\n\n### Hashing Performance (Rust, Short Words)\n\nStringZilla consistently shows strong performance in hashing, often outperforming standard libraries.\n\n\nRust:\nstringzilla::hash   ???????????????????? 1.84 GB/s\naHash::hash_one     ??????????????       1.23 GB/s\nstd::hash           ?????                0.43 GB/s\n\n\n### Case-Insensitive UTF-8 Search (Rust, English)\n\nFor Unicode-aware, case-insensitive search, StringZilla demonstrates significant speed advantages.\n\n\nRust:\n                      English\nstringzilla           ???????????????????? 12.79 GB/s\nicu                   ?                     0.08 GB/s\n\n\n### Levenshtein Distance (Rust, 1 Core)\n\nIn computationally intensive tasks like edit distance, StringZilla, especially with GPU acceleration, achieves remarkable throughput.\n\n\nRust:\n                        1 Core\nrapidfuzz               ???????????????????? 14,316 MCUPS\nstringzilla<384x GNR>   ???????????????????  13,084 MCUPS\nbio::levenshtein        ??                      823 MCUPS\n\n\n## Why Use StringWars?\n\n*   **Performance Insight**: Gain deep insights into the performance characteristics of various string processing libraries in Rust and Python.\n*   **Informed Decision-Making**: Make data-driven choices when selecting libraries for high-performance applications in areas like bioinformatics, databases, and natural language processing.\n*   **Hardware Optimization**: Understand how different libraries leverage modern CPU SIMD instructions and GPU acceleration, helping you optimize for specific hardware.\n*   **Contribution to StringZilla**: The project directly supports the development and optimization of the `StringZilla` library, offering a transparent comparison framework.\n*   **Reproducible Benchmarks**: Follow clear instructions to reproduce benchmark results on your own hardware, ensuring reliable comparisons.\n\n## Links\n\n*   **GitHub Repository**: [https://github.com/ashvardanian/StringWars](https://github.com/ashvardanian/StringWars){:target=\"_blank\"}","metrics":{"detailViews":2,"githubClicks":1},"dates":{"published":null,"modified":"2026-07-21T15:43:39.000Z"}}