Audio-AGI

AudioSep: Separate Sounds from Natural-Language Descriptions

AudioSep is a Python foundation model for separating sounds from audio based on natural-language descriptions. It supports open-domain tasks such as isolating events, instruments, or speech, with inference, fine-tuning, and evaluation workflows.

View on GitHub Compare
Stars
1.9k
Forks
157
Stars, last 30 days
+9

Our take

Health score

28/100

How it is scored
Activity
19
Community
50
Issues
21
Pull requests
20

+9 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

1,948

Forks

156

Change since Nov 16, 2026: 0 stars·0 forks

Overview

AudioSep separates a described sound source from an audio recording using a natural-language query. Rather than limiting separation to a fixed set of sound classes, it is designed for open-domain queries and reports zero-shot use across audio events, musical instruments, and speech enhancement.

The repository provides the official implementation of the paper, along with inference and training code. It is aimed at researchers and developers building or evaluating language-guided audio separation systems, rather than users looking only for a finished audio-editing application.

Key Features

  • Run source separation with a text description and an audio file.
  • Use a provided pretrained checkpoint or load a model through Hugging Face.
  • Process audio at a 32 kHz sampling rate.
  • Enable chunk-based inference to reduce memory use.
  • Fine-tune on audio-text paired datasets formatted to the supplied JSON template.
  • Evaluate separation on the provided benchmark dataset categories.
  • Explore example outputs through the project demo page.

Use Cases

  • Audio researchers can test language-guided separation across open-ended sound descriptions.
  • Music and audio developers can isolate a described instrument from a mixed recording.
  • Speech-processing teams can investigate separating speech from other sounds using text queries.
  • Teams with paired audio-text data can fine-tune the model for their own separation tasks.

Getting Started

Clone the repository and create its Conda environment:

git clone https://github.com/Audio-AGI/AudioSep.git
cd AudioSep
conda env create -f environment.yml
conda activate AudioSep

Download model weights into checkpoint/ as described in the README. The README also covers inference, training, and benchmark evaluation.

Considerations

  • Model weights are a separate download, so cloning the repository alone is not enough to run the pretrained model.
  • The documented inference setup uses 32 kHz audio. The README notes that the checkpoint's 32 kHz, 2048-point window and 320-point hop differ from the DCASE 2024 Task 9 baseline settings.
  • AudioSep is research-oriented software with model-building, training, and benchmark scripts. Users seeking a graphical audio-editing workflow may need another tool.
  • Chunk-based inference is available as a memory-saving option; the README does not specify hardware requirements.

Found this useful?

Share it with someone who would like AudioSep.