TMElyralab

MuseTalk: Generate Audio-Synced Talking-Head Videos

MuseTalk creates lip-synced video from a source video or image and an audio clip using latent-space inpainting. It supports training and inference workflows, with real-time performance reported on a Tesla V100.

View on GitHub Compare
Stars
6.7k
Forks
968
Stars, last 30 days
+233

Our take

Health score

32/100

How it is scored
Activity
19
Community
69
Issues
21
Pull requests
20

+233 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

6,975

Forks

1,022

Change since Nov 17, 2026: +10 stars·+1 forks

Overview

MuseTalk is an audio-driven lip-syncing system for modifying a face in an existing video or image so its mouth movements match supplied audio. It targets creators and developers building dubbed videos or virtual-human pipelines, including workflows that pair it with MuseV.

The model encodes image and audio features and performs a single-step latent-space inpainting operation. This is not a diffusion model, despite its use of a UNet architecture related to Stable Diffusion. The repository includes inference, real-time inference, and training code, as well as pretrained weights.

Key Features

  • Generate lip-synced video from a source video, image, or image sequence and an audio file.
  • Supports audio in languages including Chinese, English, and Japanese.
  • Provides standard and real-time inference scripts for MuseTalk 1.0 and 1.5.
  • Includes a Gradio interface for adjusting parameters and previewing a first frame.
  • Provides dataset preprocessing and staged training scripts.
  • Offers a face-region positioning control that can affect mouth openness.
  • Integrates with a virtual-human workflow using MuseV.

Use Cases

  • Video creators can dub an existing talking-head clip into another language or voice when matching mouth movement is important.
  • Virtual-human developers can add speech animation to generated character videos or still images.
  • Researchers can experiment with audio-conditioned face generation and train models using the supplied preprocessing and training workflow.
  • Teams building a video production tool can use the inference scripts or Gradio demo to evaluate lip-sync results before integrating the model.

Getting Started

The README recommends Python 3.10 and CUDA 11.7, and documents PyTorch, MMLab packages, FFmpeg, and model-weight setup. After installation and weight download, Linux users can run MuseTalk 1.5 inference with:

sh inference.sh v1.5 normal

See the README for Windows instructions, real-time inference, model downloads, and training setup. Model weights are also available from the Hugging Face repository, and a hosted demo is provided.

Alternatives

  • audio2photoreal: audio2photoreal generates a photorealistic avatar driven by audio, while MuseTalk lip-syncs an existing video or image to an audio clip.
ProjectLanguageLicenseStarsStatus
MuseTalkPythonOther6.7kNot checked yet
audio2photorealPythonOther2.8kArchived

Considerations

  • The documented setup requires a CUDA-capable GPU, several model dependencies, FFmpeg, and separate pretrained weights. The project reports 30fps+ on an NVIDIA Tesla V100, but speed depends on hardware and configuration.
  • The README notes limitations in identity preservation and temporal stability, including possible changes to facial details and jitter from single-frame generation.
  • The generated face region is 256 × 256. Higher-resolution output may require an additional super-resolution model.
  • The repository says its code is MIT-licensed, while the supplied repository metadata reports the license as NOASSERTION. Check the repository license and the terms of third-party models and datasets before use. The README restricts test data to non-commercial research purposes.

Found this useful?

Share it with someone who would like MuseTalk.

Comparisons