{"name":"MOSS-TTS Family: Open-Source High-Fidelity Speech and Sound Generation","description":"The MOSS-TTS Family offers an open-source suite of models for high-fidelity, highly expressive speech and sound generation. Designed for complex real-world scenarios, it covers stable long-form speech, multi-speaker dialogue, voice design, environmental sound effects, and real-time streaming TTS. This comprehensive family of models from MOSI.AI and OpenMOSS team provides robust solutions for diverse audio generation needs.","github":"https://github.com/OpenMOSS/MOSS-TTS","url":"https://osrepos.com/repo/openmoss-moss-tts","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/openmoss-moss-tts","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/openmoss-moss-tts.md","json":"https://osrepos.com/repo/openmoss-moss-tts.json","topics":["audio","audio-tokenizer","llm","multimodal","text-to-speech","voice-cloning","Python","speech-synthesis"],"keywords":["audio","audio-tokenizer","llm","multimodal","text-to-speech","voice-cloning","Python","speech-synthesis"],"stars":null,"summary":"The MOSS-TTS Family offers an open-source suite of models for high-fidelity, highly expressive speech and sound generation. Designed for complex real-world scenarios, it covers stable long-form speech, multi-speaker dialogue, voice design, environmental sound effects, and real-time streaming TTS. This comprehensive family of models from MOSI.AI and OpenMOSS team provides robust solutions for diverse audio generation needs.","content":"## Introduction\n\nMOSS-TTS Family, developed by MOSI.AI and the OpenMOSS team, is an open-source collection of models dedicated to advanced speech and sound generation. It is engineered to meet the demands of high-fidelity, high-expressiveness, and complex real-world applications. This family of models addresses various needs, including stable long-form speech, multi-speaker dialogue, voice and character design, environmental sound effects, and real-time streaming text-to-speech (TTS).\n\nA single audio piece often requires nuanced capabilities, such as sounding like a real person, accurate pronunciation, switching speaking styles, maintaining stability over long durations, and supporting dialogue and real-time interaction. The MOSS-TTS Family breaks down this complexity into five production-ready models that can be used independently or integrated into a complete pipeline:\n\n*   **MOSS-TTS**: The flagship model for high fidelity and optimal zero-shot voice cloning, supporting long-speech generation, fine-grained control, and multilingual synthesis.\n*   **MOSS-TTSD**: A spoken dialogue generation model for expressive, multi-speaker, and ultra-long dialogues.\n*   **MOSS-VoiceGenerator**: An open-source voice design model capable of generating diverse voices and styles directly from text prompts, without reference speech.\n*   **MOSS-TTS-Realtime**: A multi-turn context-aware model for real-time voice agents, ensuring natural and coherent replies with low latency.\n*   **MOSS-SoundEffect**: A content creation model specialized in sound effect generation with wide category coverage and controllable duration.\n\n## Installation\n\nTo get started with MOSS-TTS Family, a clean, isolated Python environment is recommended to avoid dependency conflicts. Transformers 5.0.0 is required.\n\n### Using Conda\n\n1.  Create and activate a new Conda environment:\n    bash\n    conda create -n moss-tts python=3.12 -y\n    conda activate moss-tts\n    \n2.  Clone the repository and install dependencies:\n    bash\n    git clone https://github.com/OpenMOSS/MOSS-TTS.git\n    cd MOSS-TTS\n    pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e \".[torch-runtime]\"\n    \n\n### Using `uv`\n\n1.  Install `uv` (if not already installed, see [uv documentation](https://docs.astral.sh/uv/getting-started/installation/)).\n2.  Clone the repository and create a virtual environment:\n    bash\n    git clone https://github.com/OpenMOSS/MOSS-TTS.git\n    cd MOSS-TTS\n    uv venv --python 3.12 .venv\n    source .venv/bin/activate\n    \n3.  Install dependencies:\n    bash\n    uv pip install --torch-backend cu128 -e \".[torch-runtime]\"\n    \n\n### (Optional) Install FlashAttention 2\n\nFor improved speed and reduced GPU memory usage, FlashAttention 2 can be installed if your hardware supports it.\n\nFor Conda/pip:\n\nbash\npip install --extra-index-url https://download.pytorch.org/whl/cu128 -e \".[torch-runtime,flash-attn]\"\n\n\nFor `uv`:\n\nbash\nuv pip install --torch-backend cu128 -e \".[torch-runtime,flash-attn]\"\n\n\n## Examples\n\nMOSS-TTS provides a straightforward `generate` interface. Here's a basic example demonstrating direct TTS and voice cloning:\n\npython\nfrom pathlib import Path\nimport importlib.util\nimport torch\nimport torchaudio\nfrom transformers import AutoModel, AutoProcessor\n\n# Disable the broken cuDNN SDPA backend\ntorch.backends.cuda.enable_cudnn_sdp(False)\n# Keep these enabled as fallbacks\ntorch.backends.cuda.enable_flash_sdp(True)\ntorch.backends.cuda.enable_mem_efficient_sdp(True)\ntorch.backends.cuda.enable_math_sdp(True)\n\npretrained_model_name_or_path = \"OpenMOSS-Team/MOSS-TTS-v1.5\"\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\ndtype = torch.bfloat16 if device == \"cuda\" else torch.float32\n\ndef resolve_attn_implementation() -> str:\n    if (\n        device == \"cuda\"\n        and importlib.util.find_spec(\"flash_attn\") is not None\n        and dtype in {torch.float16, torch.bfloat16}\n    ):\n        major, _ = torch.cuda.get_device_capability()\n        if major >= 8:\n            return \"flash_attention_2\"\n    if device == \"cuda\":\n        return \"sdpa\"\n    return \"eager\"\n\nattn_implementation = resolve_attn_implementation()\nprint(f\"[INFO] Using attn_implementation={attn_implementation}\")\n\nprocessor = AutoProcessor.from_pretrained(\n    pretrained_model_name_or_path,\n    trust_remote_code=True,\n)\nprocessor.audio_tokenizer = processor.audio_tokenizer.to(device)\n\ntext_chinese = \"?????\\n????\\n\\n???????????????????????????\"\ntext_english = \"We stand on the threshold of the AI era.\\nArtificial intelligence is no longer just a concept in laboratories.\"\n\n# Use audio from ./assets/audio to avoid downloading from the cloud.\nref_audio_chinese = \"https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav\"\nref_audio_english = \"https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a\"\n\nconversations = [\n    # Direct TTS (no reference). Language tags are recommended in v1.5.\n    [processor.build_user_message(text=text_chinese)],\n    [processor.build_user_message(text=text_english)],\n    # Voice cloning (with reference)\n    [processor.build_user_message(text=text_chinese, reference=[ref_audio_chinese])],\n    [processor.build_user_message(text=text_english, reference=[ref_audio_english])],\n]\n\nmodel = AutoModel.from_pretrained(\n    pretrained_model_name_or_path,\n    trust_remote_code=True,\n    attn_implementation=attn_implementation,\n    torch_dtype=dtype,\n).to(device)\nmodel.eval()\n\nsave_dir = Path(\"inference_root\")\nsave_dir.mkdir(exist_ok=True, parents=True)\nsample_idx = 0\nwith torch.no_grad():\n    for batch_conversations in conversations:\n        batch = processor([batch_conversations], mode=\"generation\")\n        input_ids = batch[\"input_ids\"].to(device)\n        attention_mask = batch[\"attention_mask\"].to(device)\n\n        outputs = model.generate(\n            input_ids=input_ids,\n            attention_mask=attention_mask,\n            max_new_tokens=4096,\n        )\n\n        for message in processor.decode(outputs):\n            audio = message.audio_codes_list[0]\n            out_path = save_dir / f\"sample{sample_idx}.wav\"\n            sample_idx += 1\n            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)\n\n\n\n## Why Use MOSS-TTS Family?\n\nMOSS-TTS Family stands out for its comprehensive approach to audio generation, offering a suite of specialized models that collectively achieve state-of-the-art performance. Key advantages include:\n\n*   **High Fidelity and Expressiveness**: The models are designed to produce highly realistic and emotionally rich speech and sound, crucial for immersive user experiences.\n*   **Versatility for Complex Scenarios**: From long-form narratives and multi-speaker dialogues to unique voice design and real-time conversational agents, MOSS-TTS Family provides tailored solutions for diverse and challenging applications.\n*   **Multilingual Support**: MOSS-TTS-v1.5 supports 31 languages, offering robust multilingual synthesis and voice cloning capabilities, making it suitable for global applications.\n*   **Optimized Performance**: With features like FlashAttention 2 support and a torch-free `llama.cpp` backend, the models are optimized for speed, lower GPU memory usage, and efficient deployment on various hardware, including CPU-only environments.\n*   **Modular Architecture**: The family's modular design allows developers to use individual models for specific tasks or combine them into powerful pipelines, offering flexibility and scalability.\n*   **Strong Evaluation Results**: MOSS-TTS models consistently achieve leading performance on objective and subjective benchmarks, often rivaling or surpassing top closed-source systems in areas like speaker attribution accuracy, voice similarity, and overall quality.\n\n## Links\n\n*   **GitHub Repository**: [https://github.com/OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS){:target=\"_blank\"}\n*   **Hugging Face Collection**: [https://huggingface.co/collections/OpenMOSS-Team/moss-tts](https://huggingface.co/collections/OpenMOSS-Team/moss-tts){:target=\"_blank\"}\n*   **ModelScope Collection**: [https://www.modelscope.cn/collections/openmoss/MOSS-TTS](https://www.modelscope.cn/collections/openmoss/MOSS-TTS){:target=\"_blank\"}\n*   **MOSS-TTS Technical Report (arXiv)**: [https://arxiv.org/abs/2603.18090](https://arxiv.org/abs/2603.18090){:target=\"_blank\"}\n*   **MOSS-TTSD Technical Report (arXiv)**: [https://arxiv.org/abs/2603.19739](https://arxiv.org/abs/2603.19739){:target=\"_blank\"}\n*   **MOSS-VoiceGenerator Technical Report (arXiv)**: [https://arxiv.org/abs/2603.28086](https://arxiv.org/abs/2603.28086){:target=\"_blank\"}","metrics":{"detailViews":13,"githubClicks":13},"dates":{"published":null,"modified":"2026-05-31T20:39:28.000Z"}}