{"name":"tair-kvcache: Manage and Simulate LLM Inference KV Caches","description":"Alibaba Cloud Tair KVCache combines global KV-cache management with HiSim, a CPU-based simulator for LLM inference performance. It is aimed at teams integrating cache storage with inference engines or evaluating workloads before allocating GPU resources.","github":"https://github.com/alibaba/tair-kvcache","url":"https://osrepos.com/repo/alibaba-tair-kvcache","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/alibaba-tair-kvcache","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/alibaba-tair-kvcache.md","json":"https://osrepos.com/repo/alibaba-tair-kvcache.json","topics":["c++","llm","distributed-systems","kv-cache","inference-simulation","cache-management"],"keywords":["c++","llm","distributed-systems","kv-cache","inference-simulation","cache-management"],"stars":null,"summary":"Alibaba Cloud Tair KVCache combines global KV-cache management with HiSim, a CPU-based simulator for LLM inference performance. It is aimed at teams integrating cache storage with inference engines or evaluating workloads before allocating GPU resources.","content":"## Overview\n\nAlibaba Cloud Tair KVCache is a high-performance KVCache system for large language model inference. Its open-source components address two needs: managing KV-cache metadata and storage across inference engines, and using HiSim, a CPU-based simulator, to estimate inference performance from workload traces without GPU resources.\n\nThe Manager and its client/connectors are intended for inference infrastructure teams integrating engines such as vLLM, SGLang, RTP-LLM, and TRT-LLM with storage backends. HiSim currently focuses on SGLang v0.5.6.post2 with Qwen3 Dense models on H20 GPUs, so its stated simulation scope is narrower than the broader Manager integrations.\n\n## Key Features\n\n- Centralized management of KV-cache metadata, queries, and storage capacity.\n- Cache matching options include prefix, sliding-window, and KV matching.\n- Two-phase writes and metadata persistence support reliable cache operations.\n- Storage abstraction supports backends including HF3FS, Mooncake, and NFS.\n- Capacity controls and background eviction manage quotas and backend water levels.\n- Client/connectors support KV-cache transfer for vLLM, SGLang, RTP-LLM, and TRT-LLM.\n- HiSim replays inference traces to estimate metrics such as TTFT, TPOT, throughput, cache hit rate, and capacity consumption.\n\n## Use Cases\n\n- Inference platform teams that need shared KV-cache metadata and capacity management across engines and storage backends.\n- Teams using supported inference engines that want to connect engine-side workloads to KV-cache storage through the project’s client/connectors.\n- Infrastructure planners who want to replay workload traces with HiSim before committing GPU resources, within its documented model and engine scope.\n- Operators tuning cache capacity or eviction settings who want to compare simulated cache behavior and resource use.\n\n## Our Take\n\nTair KVCache is an actively developed infrastructure project with a substantial contributor base, but its components have different adoption scopes and HiSim’s documented support is specific.\n\n**Good fit if:**\n- You are building LLM inference infrastructure and need centralized KV-cache metadata, capacity controls, or storage integration.\n- Your engine is among the listed connector targets, or your simulation workload matches HiSim’s documented SGLang and Qwen3 Dense setup.\n- You can evaluate a C++ project with CI and recent releases, and are prepared to check the available test coverage yourself.\n\n**Look elsewhere if:**\n- You need a general-purpose inference simulator with documented support across engines, models, or hardware beyond HiSim’s stated scope.\n- You need a ready-to-use end-user cache product rather than components to integrate into inference infrastructure.\n- Your deployment depends on a specific storage backend or engine that is not covered by the repository’s stated integrations.\n\n## Project Health\n\n| Signal | Value |\n|---|---|\n| Status | **Active**: regular commits and releases |\n| Last commit | 2026-10-28 (today) |\n| Commits, last 90 days | 95 |\n| Releases, last 12 months | 2 (latest __binary-dependency-0.0.2, 2026-02-12, pre-release) |\n| Contributors | 24 (top contributor: 30% of commits) |\n| Issues closed, last 90 days | 0+ |\n| Pull requests merged, last 90 days | 70 (typically merged in 1 day) |\n| Project age | 10 months |\n\nChecked on 2026-10-28 with the GitHub API.\n\n## Project Facts\n\n- Language: C++\n- License: Apache-2.0\n- Stars: 275\n- Forks: 65\n- Topics: hisim, kv-cache, kvcache, llm, simulator\n- Archived: no\n\n## What You Need\n\nDetected in the repository:\n\n- C++ (main language on GitHub)\n- Automated checks on GitHub Actions\n\n## Getting Started\n\nStart with the [repository README](https://github.com/alibaba/tair-kvcache) to choose between the Manager and HiSim and follow their component-specific setup instructions. For HiSim details, see the [HiSim README](https://github.com/alibaba/tair-kvcache/blob/main/hisim/README.md).\n\n## License in Plain Words\n\n**Apache-2.0** (permissive).\n\n- Commercial use: yes\n- Modify and redistribute: yes\n- You must keep: the license, the NOTICE file and a note of your changes\n- Share your changes: no\n- Includes an explicit patent grant from the contributors\n\nA summary, not legal advice: the LICENSE file is what applies.\n\n## Considerations\n\nThe repository is active, with regular commits and releases over the past year, and development is spread across multiple contributors rather than concentrated in one person. However, the supplied project data does not indicate a detected test suite, so teams should validate the components and workflows they intend to deploy. HiSim’s documented configuration is specific to SGLang v0.5.6.post2, Qwen3 Dense models, and H20 GPUs; confirm that this scope matches your workload before relying on its estimates. The Manager’s usefulness also depends on integrating its clients and storage abstractions with your chosen inference and storage systems.","metrics":{"detailViews":3,"githubClicks":0},"dates":{"published":null,"modified":"2026-10-28T16:08:25.000Z"}}