{"name":"chatterbox-vllm: Accelerating Chatterbox TTS with vLLM for Enhanced Performance","description":"chatterbox-vllm is a high-performance port of the Chatterbox Text-to-Speech (TTS) model to vLLM, designed to significantly improve generation speed and GPU memory efficiency. This personal project aims to provide a more efficient and easily integratable solution for speech synthesis, offering substantial speedups compared to the original implementation. While currently usable and demonstrating benchmark-topping throughput, it leverages internal vLLM APIs and hacky workarounds, with ongoing refactoring planned.","github":"https://github.com/randombk/chatterbox-vllm","url":"https://osrepos.com/repo/randombk-chatterbox-vllm","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/randombk-chatterbox-vllm","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/randombk-chatterbox-vllm.md","json":"https://osrepos.com/repo/randombk-chatterbox-vllm.json","topics":["Python","TTS","vLLM","Machine Learning","AI","Deep Learning","Speech Synthesis","Performance"],"keywords":["Python","TTS","vLLM","Machine Learning","AI","Deep Learning","Speech Synthesis","Performance"],"stars":null,"summary":"chatterbox-vllm is a high-performance port of the Chatterbox Text-to-Speech (TTS) model to vLLM, designed to significantly improve generation speed and GPU memory efficiency. This personal project aims to provide a more efficient and easily integratable solution for speech synthesis, offering substantial speedups compared to the original implementation. While currently usable and demonstrating benchmark-topping throughput, it leverages internal vLLM APIs and hacky workarounds, with ongoing refactoring planned.","content":"## Introduction\n\n`chatterbox-vllm` is an impressive project that ports the Chatterbox Text-to-Speech (TTS) model to vLLM, a high-performance inference engine. Developed by randombk, this repository aims to dramatically enhance the performance and efficiency of the Chatterbox model, making it faster and more memory-friendly on GPUs. It's a personal project focused on leveraging vLLM's capabilities for state-of-the-art speech synthesis. Early benchmarks indicate significant speedups, making it an exciting development for anyone working with TTS models.\n\n## Installation\n\nThis project primarily supports Linux and WSL2 with Nvidia hardware. While AMD might work with minor adjustments, it has not been tested.\n\nPrerequisites: Ensure `git` and [`uv`](https://pypi.org/project/uv/) (a fast Python package installer and resolver) are installed on your system.\n\nbash\ngit clone https://github.com/randombk/chatterbox-vllm.git\ncd chatterbox-vllm\nuv venv\nsource .venv/bin/activate\nuv sync\n\n\nThe necessary model weights should be automatically downloaded from the Hugging Face Hub. If you encounter CUDA-related issues, try resetting your virtual environment and using `uv pip install -e .` instead of `uv sync`.\n\n## Examples\n\nTo quickly generate audio samples, you can run the provided `example-tts.py` script. This example demonstrates how to generate speech for multiple prompts using different voices.\n\npython\nimport torchaudio as ta\nfrom chatterbox_vllm.tts import ChatterboxTTS\n\n\nif __name__ == \"__main__\":\n    model = ChatterboxTTS.from_pretrained(\n        gpu_memory_utilization = 0.4,\n        max_model_len = 1000,\n\n        # Disable CUDA graphs to reduce startup time for one-off generation.\n        enforce_eager = True,\n    )\n\n    for i, audio_prompt_path in enumerate([None, \"docs/audio-sample-01.mp3\", \"docs/audio-sample-03.mp3\"]):\n        prompts = [\n            \"You are listening to a demo of the Chatterbox TTS model running on VLLM.\",\n            \"This is a separate prompt to test the batching implementation.\",\n            \"And here is a third prompt. It's a bit longer than the first one, but not by much.\",\n        ]\n    \n        audios = model.generate(prompts, audio_prompt_path=audio_prompt_path, exaggeration=0.8)\n        for audio_idx, audio in enumerate(audios):\n            ta.save(f\"test-{i}-{audio_idx}.mp3\", audio, model.sr)\n\n\n## Why Use It\n\nThe primary motivation behind `chatterbox-vllm` is to overcome performance bottlenecks and improve GPU memory utilization of the original Chatterbox TTS model. By porting it to vLLM, the project achieves:\n\n*   **Improved Performance**: Early benchmarks show significant speedups, with generation tokens/s increasing by approximately 4x without batching and over 10x with batching. This is a substantial improvement over the original implementation, which was often bottlenecked by CPU-GPU synchronization.\n*   **Efficient GPU Memory Use**: vLLM's optimized inference infrastructure allows for more efficient use of GPU memory, enabling higher throughput and potentially larger batch sizes.\n*   **Easier Integration**: The vLLM port facilitates easier integration with modern, high-performance inference systems, streamlining deployment and scaling of TTS applications.\n*   **Benchmark-Topping Throughput**: The project currently boasts impressive throughput, particularly for the T3 Llama token generation component, which is no longer the bottleneck in the TTS pipeline.\n\n## Links\n\n*   **GitHub Repository**: [https://github.com/randombk/chatterbox-vllm](https://github.com/randombk/chatterbox-vllm)","metrics":{"detailViews":5,"githubClicks":4},"dates":{"published":null,"modified":"2025-10-11T21:05:50.000Z"}}