AudioSep: Separate Sounds from Natural-Language Descriptions

Overview

AudioSep separates a described sound source from an audio recording using a natural-language query. Rather than limiting separation to a fixed set of sound classes, it is designed for open-domain queries and reports zero-shot use across audio events, musical instruments, and speech enhancement.

The repository provides the official implementation of the paper, along with inference and training code. It is aimed at researchers and developers building or evaluating language-guided audio separation systems, rather than users looking only for a finished audio-editing application.

Key Features

• Run source separation with a text description and an audio file.
• Use a provided pretrained checkpoint or load a model through Hugging Face.
• Process audio at a 32 kHz sampling rate.
• Enable chunk-based inference to reduce memory use.
• Fine-tune on audio-text paired datasets formatted to the supplied JSON template.
• Evaluate separation on the provided benchmark dataset categories.
• Explore example outputs through the project demo page.

Use Cases

• Audio researchers can test language-guided separation across open-ended sound descriptions.
• Music and audio developers can isolate a described instrument from a mixed recording.
• Speech-processing teams can investigate separating speech from other sounds using text queries.
• Teams with paired audio-text data can fine-tune the model for their own separation tasks.

Project Facts

• Language: Python
• License: MIT
• Stars: 1.9k
• Forks: 157
• Topics: none listed
• Archived: No

Getting Started

Clone the repository and create its Conda environment:

git clone https://github.com/Audio-AGI/AudioSep.git
cd AudioSep
conda env create -f environment.yml
conda activate AudioSep

Download model weights into checkpoint/ as described in the README. The README also covers inference, training, and benchmark evaluation.

Considerations

• Model weights are a separate download, so cloning the repository alone is not enough to run the pretrained model.
• The documented inference setup uses 32 kHz audio. The README notes that the checkpoint's 32 kHz, 2048-point window and 320-point hop differ from the DCASE 2024 Task 9 baseline settings.
• AudioSep is research-oriented software with model-building, training, and benchmark scripts. Users seeking a graphical audio-editing workflow may need another tool.
• Chunk-based inference is available as a memory-saving option; the README does not specify hardware requirements.