# chatterbox-vllm: Accelerating Chatterbox TTS with vLLM for Enhanced Performance

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/randombk-chatterbox-vllm
Generated for open source discovery and AI-assisted research.

chatterbox-vllm is a high-performance port of the Chatterbox Text-to-Speech (TTS) model to vLLM, designed to significantly improve generation speed and GPU memory efficiency. This personal project aims to provide a more efficient and easily integratable solution for speech synthesis, offering substantial speedups compared to the original implementation. While currently usable and demonstrating benchmark-topping throughput, it leverages internal vLLM APIs and hacky workarounds, with ongoing refactoring planned.

GitHub: https://github.com/randombk/chatterbox-vllm
OSRepos URL: https://osrepos.com/repo/randombk-chatterbox-vllm

## Summary

chatterbox-vllm is a high-performance port of the Chatterbox Text-to-Speech (TTS) model to vLLM, designed to significantly improve generation speed and GPU memory efficiency. This personal project aims to provide a more efficient and easily integratable solution for speech synthesis, offering substantial speedups compared to the original implementation. While currently usable and demonstrating benchmark-topping throughput, it leverages internal vLLM APIs and hacky workarounds, with ongoing refactoring planned.

## Topics

- Python
- TTS
- vLLM
- Machine Learning
- AI
- Deep Learning
- Speech Synthesis
- Performance

## Repository Information

Last analyzed by OSRepos: Sat Oct 11 2025 22:05:50 GMT+0100 (Western European Summer Time)
Detail views: 5
GitHub clicks: 4

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

`chatterbox-vllm` is an impressive project that ports the Chatterbox Text-to-Speech (TTS) model to vLLM, a high-performance inference engine. Developed by randombk, this repository aims to dramatically enhance the performance and efficiency of the Chatterbox model, making it faster and more memory-friendly on GPUs. It's a personal project focused on leveraging vLLM's capabilities for state-of-the-art speech synthesis. Early benchmarks indicate significant speedups, making it an exciting development for anyone working with TTS models.

## Installation

This project primarily supports Linux and WSL2 with Nvidia hardware. While AMD might work with minor adjustments, it has not been tested.

Prerequisites: Ensure `git` and [`uv`](https://pypi.org/project/uv/) (a fast Python package installer and resolver) are installed on your system.

bash
git clone https://github.com/randombk/chatterbox-vllm.git
cd chatterbox-vllm
uv venv
source .venv/bin/activate
uv sync


The necessary model weights should be automatically downloaded from the Hugging Face Hub. If you encounter CUDA-related issues, try resetting your virtual environment and using `uv pip install -e .` instead of `uv sync`.

## Examples

To quickly generate audio samples, you can run the provided `example-tts.py` script. This example demonstrates how to generate speech for multiple prompts using different voices.

python
import torchaudio as ta
from chatterbox_vllm.tts import ChatterboxTTS


if __name__ == "__main__":
    model = ChatterboxTTS.from_pretrained(
        gpu_memory_utilization = 0.4,
        max_model_len = 1000,

        # Disable CUDA graphs to reduce startup time for one-off generation.
        enforce_eager = True,
    )

    for i, audio_prompt_path in enumerate([None, "docs/audio-sample-01.mp3", "docs/audio-sample-03.mp3"]):
        prompts = [
            "You are listening to a demo of the Chatterbox TTS model running on VLLM.",
            "This is a separate prompt to test the batching implementation.",
            "And here is a third prompt. It's a bit longer than the first one, but not by much.",
        ]
    
        audios = model.generate(prompts, audio_prompt_path=audio_prompt_path, exaggeration=0.8)
        for audio_idx, audio in enumerate(audios):
            ta.save(f"test-{i}-{audio_idx}.mp3", audio, model.sr)


## Why Use It

The primary motivation behind `chatterbox-vllm` is to overcome performance bottlenecks and improve GPU memory utilization of the original Chatterbox TTS model. By porting it to vLLM, the project achieves:

*   **Improved Performance**: Early benchmarks show significant speedups, with generation tokens/s increasing by approximately 4x without batching and over 10x with batching. This is a substantial improvement over the original implementation, which was often bottlenecked by CPU-GPU synchronization.
*   **Efficient GPU Memory Use**: vLLM's optimized inference infrastructure allows for more efficient use of GPU memory, enabling higher throughput and potentially larger batch sizes.
*   **Easier Integration**: The vLLM port facilitates easier integration with modern, high-performance inference systems, streamlining deployment and scaling of TTS applications.
*   **Benchmark-Topping Throughput**: The project currently boasts impressive throughput, particularly for the T3 Llama token generation component, which is no longer the bottleneck in the TTS pipeline.

## Links

*   **GitHub Repository**: [https://github.com/randombk/chatterbox-vllm](https://github.com/randombk/chatterbox-vllm)