csm: Generate Conversational Speech from Text and Audio

Overview

CSM generates speech audio from text, with optional audio-and-transcript context to shape a response in a conversation. Its architecture combines a Llama backbone with an audio decoder that produces Mimi audio codes.

It is a speech-generation component, not a general-purpose conversational assistant: applications that need generated text should pair it with a separate language model. The repository provides code for running the 1B model, while the checkpoint is hosted separately on Hugging Face.

Key Features

• Generates audio from text with a selected speaker identity.
• Accepts prior utterances as context using transcript, speaker, and audio segments.
• Supports conversational generation involving multiple speaker identities.
• Uses a Llama-based backbone and Mimi audio codes.
• Provides Python examples and a quickstart script for local generation.
• Available through Hugging Face Transformers and a hosted demo, as noted by the project.

Use Cases

• Developers prototyping voice interfaces that generate spoken responses from application-provided text.
• Researchers exploring speech generation conditioned on conversational audio and transcripts.
• Teams generating dialogue for multiple speaker identities, when they can supply suitable context and audio infrastructure.
• Builders pairing a separate text-generating LLM with speech output for an interactive voice application.

Getting Started

The repository recommends Python 3.10 and requires access to the CSM-1B and Llama-3.2-1B Hugging Face models. A minimal setup is:

git clone https://github.com/SesameAILabs/csm.git
cd csm
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
export NO_TORCH_COMPILE=1
huggingface-cli login
python run_csm.py

See the repository README for usage examples and platform-specific setup details.

Alternatives

• MOSS-TTS: MOSS-TTS also generates speech, but spans long-form narration, voice design, sound effects, and streaming across multiple model architectures.
• VoxCPM: VoxCPM focuses on multilingual, tokenizer-free speech synthesis and voice cloning, with VoxCPM2 requiring support for a 2B-parameter model.
• Spark-TTS: Spark-TTS is a bilingual PyTorch inference project built on Qwen2.5, with zero-shot voice cloning and adjustable voice characteristics.
• Chatterbox: Chatterbox offers general-purpose and multilingual TTS models, including a Turbo variant with paralinguistic tags.

Considerations

• The documented requirements call for a CUDA-compatible GPU; the README notes testing with CUDA 12.4 and 12.6. Model inference is resource-intensive compared with ordinary Python utilities.
• Running the examples requires access to both the CSM-1B and Llama-3.2-1B checkpoints on Hugging Face. Some audio operations may also require ffmpeg.
• The open model is a base generation model, not a voice fine-tuned for a particular identity. It generates audio, not text, and the README cautions that non-English performance is likely limited.
• Sesame explicitly prohibits uses such as non-consensual impersonation, fraud, and harmful or illegal activity. Review the project’s usage guidance before deployment.
• For further integration options, the README notes native Transformers support as of version 4.52.1.

Repository: SesameAILabs/csm