NVIDIA NeMo Speech: Scalable Generative AI for Speech Models

Introduction

NVIDIA NeMo Speech is a comprehensive generative AI framework tailored for researchers and PyTorch developers working on advanced speech models. This includes Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Speech Large Language Models (LLMs). The framework is engineered to streamline the process of creating, customizing, and deploying new AI models by leveraging existing code and a rich collection of pre-trained model checkpoints. It supports a wide range of applications, from real-time streaming ASR to high-quality multilingual TTS.

Installation

NeMo Speech offers flexible installation options, allowing users to integrate it with their preferred Python, PyTorch, and CUDA versions.

From source with uv (recommended)

For a reproducible setup with the actively-tested stack (Python 3.13, PyTorch 2.12, CUDA 13.2), uv is the recommended installer.

git clone https://github.com/NVIDIA-NeMo/NeMo.git
cd NeMo
uv sync --extra all --extra cu13     # For CUDA 13.x (recommended)
# Use --extra cu12 for CUDA 12.x

This installs NeMo into a .venv/ in editable mode.

From PyPI with pip (fallback, bring your own versions)

If you prefer to use your own Python/PyTorch/CUDA stack, install PyTorch first (version ? 2.7), then install nemo-toolkit via pip or uv pip.

uv pip install 'nemo-toolkit[asr,tts]'   # or plain: pip install 'nemo-toolkit[asr,tts]'

To pull NVIDIA's pinned PyTorch build, add the CUDA extra and the matching wheel index:

pip install 'nemo-toolkit[asr,tts,cu13]' --extra-index-url https://download.pytorch.org/whl/cu132   # CUDA 13.x
pip install 'nemo-toolkit[asr,tts,cu12]' --extra-index-url https://download.pytorch.org/whl/cu126   # CUDA 12.x

Examples

NVIDIA NeMo Speech provides a rich set of pre-trained models and demos to get started quickly. You can explore various models for ASR, TTS, and Speech LLMs.

• HuggingFace Collection: Discover the latest open-weight checkpoints and interactive demos on the official NVIDIA NeMo HuggingFace collection.
• Documentation: The NeMo Framework User Guide offers comprehensive technical documentation, including tutorials and examples for implementing different speech models.
• Specific Demos: Look for demos like Nemotron 3 VoiceChat for full-duplex conversations or Nemotron-Speech-Streaming for real-time ASR.

Why Use NVIDIA NeMo Speech?

NVIDIA NeMo Speech stands out as a robust framework for several reasons:

• Scalability: Built for large-scale generative AI, it supports complex models and high-throughput applications.
• Comprehensive Features: It covers a broad spectrum of speech AI tasks, including ASR, TTS, and multimodal LLMs.
• Efficiency: Designed to help researchers and developers efficiently create, customize, and deploy new AI models.
• Pre-trained Models: Access to a vast collection of pre-trained model checkpoints accelerates development and experimentation.
• Flexibility: Works with your chosen Python, PyTorch, and CUDA versions, offering adaptability to existing environments.
• Active Development: The project is actively maintained by NVIDIA, ensuring continuous updates and support for cutting-edge research.

Links

• GitHub Repository: https://github.com/NVIDIA-NeMo/Speech
• Official Documentation: https://docs.nvidia.com/nemo/speech/nightly/
• HuggingFace Collection: https://huggingface.co/collections/nvidia/nemotron-speech
• Contributing Guide: https://github.com/NVIDIA-NeMo/NeMo/blob/main/CONTRIBUTING.md