{"name":"CSM: A Conversational Speech Generation Model by SesameAILabs","description":"CSM (Conversational Speech Model) is an advanced speech generation model from SesameAILabs, designed to create RVQ audio codes from text and audio inputs. It leverages a Llama backbone and a smaller audio decoder for Mimi audio codes, enabling high-quality, context-aware speech synthesis. The model is now natively available in Hugging Face Transformers, making it accessible for researchers and developers.","github":"https://github.com/SesameAILabs/csm","url":"https://osrepos.com/repo/sesameailabs-csm","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/sesameailabs-csm","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/sesameailabs-csm.md","json":"https://osrepos.com/repo/sesameailabs-csm.json","topics":["Python","AI","Speech Generation","Machine Learning","Conversational AI","Deep Learning","Hugging Face"],"keywords":["Python","AI","Speech Generation","Machine Learning","Conversational AI","Deep Learning","Hugging Face"],"stars":null,"summary":"CSM (Conversational Speech Model) is an advanced speech generation model from SesameAILabs, designed to create RVQ audio codes from text and audio inputs. It leverages a Llama backbone and a smaller audio decoder for Mimi audio codes, enabling high-quality, context-aware speech synthesis. The model is now natively available in Hugging Face Transformers, making it accessible for researchers and developers.","content":"## Introduction\n\nCSM, or Conversational Speech Model, is a cutting-edge speech generation model developed by [SesameAILabs](https://www.sesame.com/ \"SesameAILabs website\" target=\"_blank\"). This model specializes in generating RVQ audio codes from both text and audio inputs, making it highly versatile for various speech synthesis applications. Its architecture is built upon a robust [Llama](https://www.llama.com/ \"Llama website\" target=\"_blank\") backbone, complemented by a smaller audio decoder that produces [Mimi](https://huggingface.co/kyutai/mimi \"Mimi model on Hugging Face\" target=\"_blank\") audio codes.\n\nCSM is the technology powering the interactive voice demo showcased in Sesame's [research blog post](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice \"Sesame research blog post\" target=\"_blank\"). As of version `4.52.1`, CSM is natively available in [Hugging Face Transformers](https://huggingface.co/docs/transformers/main/en/model_doc/csm \"Hugging Face Transformers documentation for CSM\" target=\"_blank\"), simplifying its integration into existing projects. For quick testing, a hosted [Hugging Face Space](https://huggingface.co/spaces/sesame/csm-1b \"CSM-1B Hugging Face Space\" target=\"_blank\") is also available.\n\n## Installation\n\nTo get started with CSM, ensure you meet the following requirements:\n\n*   A CUDA-compatible GPU.\n*   The code has been tested on CUDA 12.4 and 12.6, but may work on other versions.\n*   Python 3.10 is recommended, though newer versions might also be compatible.\n*   `ffmpeg` may be required for certain audio operations.\n*   Access to the following Hugging Face models:\n    *   [Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B \"Llama-3.2-1B on Hugging Face\" target=\"_blank\")\n    *   [CSM-1B](https://huggingface.co/sesame/csm-1b \"CSM-1B on Hugging Face\" target=\"_blank\")\n\nFollow these steps for setup:\n\nbash\ngit clone git@github.com:SesameAILabs/csm.git\ncd csm\npython3.10 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n\n# Disable lazy compilation in Mimi\nexport NO_TORCH_COMPILE=1\n\n# You will need access to CSM-1B and Llama-3.2-1B\nhuggingface-cli login\n\n\nFor Windows users, please note that the `triton` package cannot be installed directly. Instead, use `pip install triton-windows`.\n\n## Examples\n\n### Quickstart\n\nTo generate a conversation between two characters using prompts, run the following script:\n\nbash\npython run_csm.py\n\n\n### Generate a Sentence\n\nThis example demonstrates how to generate a single sentence using a random speaker identity, as no prompt or context is provided.\n\npython\nfrom generator import load_csm_1b\nimport torchaudio\nimport torch\n\nif torch.backends.mps.is_available():\n    device = \"mps\"\nelif torch.cuda.is_available():\n    device = \"cuda\"\nelse:\n    device = \"cpu\"\n\ngenerator = load_csm_1b(device=device)\n\naudio = generator.generate(\n    text=\"Hello from Sesame.\",\n    speaker=0,\n    context=[],\n    max_audio_length_ms=10_000,\n)\n\ntorchaudio.save(\"audio.wav\", audio.unsqueeze(0).cpu(), generator.sample_rate)\n\n\n### Generate with Context\n\nCSM performs optimally when provided with context. You can prompt or provide context to the model using a `Segment` for each speaker's utterance. The following example is instructional and assumes audio files exist.\n\npython\nfrom generator import Segment\n\nspeakers = [0, 1, 0, 0]\ntranscripts = [\n    \"Hey how are you doing.\",\n    \"Pretty good, pretty good.\",\n    \"I'm great.\",\n    \"So happy to be speaking to you.\",\n]\naudio_paths = [\n    \"utterance_0.wav\",\n    \"utterance_1.wav\",\n    \"utterance_2.wav\",\n    \"utterance_3.wav\",\n]\n\ndef load_audio(audio_path):\n    audio_tensor, sample_rate = torchaudio.load(audio_path)\n    audio_tensor = torchaudio.functional.resample(\n        audio_tensor.squeeze(0), orig_freq=sample_rate, new_freq=generator.sample_rate\n    )\n    return audio_tensor\n\nsegments = [\n    Segment(text=transcript, speaker=speaker, audio=load_audio(audio_path))\n    for transcript, speaker, audio_path in zip(transcripts, speakers, audio_paths)\n]\naudio = generator.generate(\n    text=\"Me too, this is some cool stuff huh?\",\n    speaker=1,\n    context=segments,\n    max_audio_length_ms=10_000,\n)\n\ntorchaudio.save(\"audio.wav\", audio.unsqueeze(0).cpu(), generator.sample_rate)\n\n\n## Why Use CSM?\n\nCSM offers a powerful solution for high-quality conversational speech generation, making it an excellent tool for research and educational purposes. Its ability to leverage context allows for more natural and coherent speech synthesis, which is crucial for creating engaging audio experiences.\n\nIt's important to note that the open-sourced model is a base generation model. While it can produce a variety of voices, it has not been fine-tuned on any specific voice. CSM is designed as an audio generation model, not a general-purpose multimodal LLM, meaning it does not generate text. For text generation, a separate LLM is recommended. Although the model may have some capacity for non-English languages due to training data contamination, its performance in such languages is likely to be limited.\n\n### Misuse and Abuse ??\n\nThis project provides a high-quality speech generation model for research and educational purposes. While responsible and ethical use is encouraged, the following activities are **explicitly prohibited**:\n\n*   **Impersonation or Fraud**: Do not use this model to generate speech that mimics real individuals without their explicit consent.\n*   **Misinformation or Deception**: Do not use this model to create deceptive or misleading content, such as fake news or fraudulent calls.\n*   **Illegal or Harmful Activities**: Do not use this model for any illegal, harmful, or malicious purposes.\n\nBy using this model, you agree to comply with all applicable laws and ethical guidelines. We are **not responsible** for any misuse, and we strongly condemn unethical applications of this technology.\n\n## Links\n\n*   **GitHub Repository**: [https://github.com/SesameAILabs/csm](https://github.com/SesameAILabs/csm \"CSM GitHub Repository\" target=\"_blank\")\n*   **Hugging Face Transformers Documentation**: [https://huggingface.co/docs/transformers/main/en/model_doc/csm](https://huggingface.co/docs/transformers/main/en/model_doc/csm \"CSM Hugging Face Transformers Documentation\" target=\"_blank\")\n*   **CSM-1B Model on Hugging Face**: [https://huggingface.co/sesame/csm-1b](https://huggingface.co/sesame/csm-1b \"CSM-1B Model on Hugging Face\" target=\"_blank\")\n*   **Sesame Interactive Voice Demo**: [https://www.sesame.com/voicedemo](https://www.sesame.com/voicedemo \"Sesame Interactive Voice Demo\" target=\"_blank\")\n*   **Sesame Research Blog Post**: [https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice \"Sesame Research Blog Post\" target=\"_blank\")\n*   **Hugging Face Space**: [https://huggingface.co/spaces/sesame/csm-1b](https://huggingface.co/spaces/sesame/csm-1b \"CSM-1B Hugging Face Space\" target=\"_blank\")","metrics":{"detailViews":9,"githubClicks":9},"dates":{"published":null,"modified":"2026-01-11T20:01:05.000Z"}}