SyncTalk: Generate Synchronized Talking-Head Videos

Overview

SyncTalk is a research implementation for audio-driven talking-head synthesis. It uses tri-plane hash representations to preserve a subject’s identity while generating synchronized speech-related mouth movements, facial expressions, and head poses, with hair-detail restoration for high-resolution output.

It is aimed at researchers and developers who want to reproduce or adapt a subject-specific talking-head pipeline, rather than users seeking a lightweight general-purpose video editor. The project includes pretrained-model inference, training, and preprocessing for a new video. See the paper and project page for research context and demonstrations.

Key Features

• Generate talking-head video conditioned on target audio.
• Model synchronized lip motion, facial expression, and head pose.
• Preserve subject identity using tri-plane hash representations.
• Run inference with provided pretrained model files or train a subject-specific model.
• Preprocess source footage, including face tracking and audio-feature extraction.
• Choose AVE, DeepSpeech, or HuBERT audio features.
• Optionally composite the generated face onto the original image with portrait mode.
• Includes a Windows integration package as well as a Linux setup path.

Use Cases

• Researchers reproducing or extending CVPR 2024 work on audio-driven talking-head synthesis.
• Developers testing a subject-specific digital presenter from suitable portrait footage and speech audio.
• Teams comparing audio feature encoders or portrait compositing approaches in a research pipeline.
• Creators evaluating generated speech animation for a prepared character, when they can meet the data and compute requirements.

Getting Started

For Linux, the README describes a tested setup using Ubuntu 18.04, Python 3.8.8, PyTorch 1.12.1, and CUDA 11.3. After installing the listed dependencies and preparing the provided model and data files, a sample evaluation can be run with:

python main.py data/May --workspace model/trial_may -O --test --asr_model ave

Start with the README for dependency installation, data preparation, training, inference, and Windows instructions. A Google Colab notebook is also linked there.

Alternatives

• InfiniteTalk: InfiniteTalk also synchronizes talking videos to audio, with support for image or video input and long-form generation.
• HunyuanVideo-Avatar: HunyuanVideo-Avatar generates emotion-controllable videos from avatar images and audio, including scenes with multiple characters.

Considerations

• This is a research pipeline with substantial setup: preprocessing requires face-parsing and 3D morphable-model assets, and the README specifies CUDA-compatible dependencies.
• New-video preprocessing expects footage at 25 FPS, around 512×512 resolution, and approximately 4–5 minutes long.
• Training is subject-specific and can require significant memory; the README notes that preloading data to CPU or GPU is memory-intensive.
• The README notes that MediaPipe blendshape capture may not work well for some inputs and suggests OpenFace AU45 as an alternative for eye parameters.
• Torso training can address a double-chin artifact, but it does not support portrait mode.
• The repository points users seeking faster processing and better visual quality to the separate SyncTalk_2D project.
• The license field is reported as NOASSERTION, so review the repository’s licensing terms before relying on it in a project.