riffusion

riffusion-hobby: Generate Music Audio with Stable Diffusion

Riffusion-hobby is a Python library and application for generating music and audio with stable diffusion, including prompt interpolation and spectrogram-to-audio conversion. It suits developers and experimenters who want to run inference locally, but is no longer actively maintained.

View on GitHub Compare
Stars
3.9k
Forks
476
Stars, last 30 days
+9

Our take

Health score

22/100

How it is scored
Activity
19
Community
28
Issues
21
Pull requests
20

+9 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

3,908

Forks

479

Change since Nov 14, 2026: 0 stars·0 forks

Overview

Riffusion-hobby applies stable diffusion to real-time music and audio generation. It connects text prompts and image conditioning with spectrogram processing, then converts generated spectrogram images into audio clips.

The repository is the core processing code, with a command-line interface, an interactive Streamlit playground, and a Flask inference API. It may fit developers exploring diffusion-based audio or building on the Riffusion workflow, rather than users seeking a currently maintained product.

Key Features

  • Interpolates between prompts and uses image conditioning in a diffusion pipeline.
  • Converts spectrogram images to audio clips and audio clips to spectrogram images.
  • Provides a command-line interface for common processing tasks.
  • Includes a Streamlit playground for interactive use.
  • Serves model inference through a Flask API.
  • Supports CPU, CUDA, and Apple Silicon MPS inference, with different performance and compatibility caveats.
  • Allows a custom checkpoint directory or Hugging Face ID in diffusers format.

Use Cases

  • Developers can experiment with generating audio from text prompts and interpolating between musical ideas.
  • Researchers and hobbyists can explore how diffusion models connect spectrogram images with audio.
  • Application builders can run the Flask inference server locally and connect it to the related Riffusion web app.
  • Python users can convert existing audio and spectrogram files through the CLI or library code.

Getting Started

The README documents a Python 3.9 setup and dependency installation:

python -m pip install -r requirements.txt

For installation details, optional audio-format support, and setup of the model and backends, see the repository README.

Alternatives

  • Magenta RT: Magenta RT generates music from text and audio prompts in real time, while riffusion-hobby uses stable diffusion with spectrogram-to-audio conversion and prompt interpolation.
ProjectLanguageLicenseStarsStatus
riffusion-hobbyPythonMIT3.9kNot checked yet
magenta-realtimePythonApache-2.01.8kActive

Considerations

The README states that the project is no longer actively maintained. CPU inference is supported but described as quite slow. Real-time generation requires a GPU capable of running stable diffusion in approximately five seconds, and MPS may fall back to CPU for some audio operations and is not deterministic. Audio formats beyond WAV require ffmpeg; the documented setup was tested with Python 3.9 and 3.10.

Found this useful?

Share it with someone who would like riffusion-hobby.

Comparisons