SyncTalk: High-Quality Talking Head Synthesis from CVPR 2024

Introduction

SyncTalk is an innovative project presented at CVPR 2024, offering a robust solution for talking head synthesis. It aims to generate highly synchronized talking head videos, ensuring natural lip movements, facial expressions, and stable head poses. The system employs tri-plane hash representations to preserve subject identity and can restore intricate hair details, producing high-resolution video outputs. For even faster and better visual quality, consider exploring the related SyncTalk_2D project.

Installation

SyncTalk supports both Windows and Linux environments.

For Windows

A pre-built integration package is available. Download SyncTalk-Windows.zip and unzip it, then double-click inference.bat to run the demo.

Download links:

• Hugging Face
• Baidu Netdisk (pwd: 6666)

For Linux

Tested on Ubuntu 18.04, Pytorch 1.12.1 and CUDA 11.3.

First, clone the repository:

git clone https://github.com/ZiqiaoPeng/SyncTalk.git
cd SyncTalk

Then, set up the environment and install dependencies:

conda create -n synctalk python==3.8.8
conda activate synctalk
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
sudo apt-get install portaudio19-dev
pip install -r requirements.txt
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1121/download.html
pip install tensorflow-gpu==2.8.1
pip install ./freqencoder ./shencoder ./gridencoder ./raymarching

If PyTorch3D installation encounters issues, use:

python ./scripts/install_pytorch3d.py

Data Preparation:

Download pre-trained models (May.zip to data/, trial_may.zip to model/) and unzip them. For processing your own video, prepare face-parsing and 3DMM models by running provided wget commands and convert_BFM.py script. Place your video (data/<ID>/<ID>.mp4) and process it:

python data_utils/process.py data/<ID>/<ID>.mp4 --asr ave

Optionally, obtain AU45 for eye blinking using OpenFace.

Examples

Run Evaluation

Evaluate the model with or without portrait mode:

python main.py data/May --workspace model/trial_may -O --test --asr_model ave
python main.py data/May --workspace model/trial_may -O --test --asr_model ave --portrait

Inference with Target Audio

Generate talking head video from a .wav audio file:

python main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav

You can also use DeepSpeech or HuBERT for audio features.

Training

Train a new model or fine-tune an existing one:

python main.py data/May --workspace model/trial_may -O --iters 60000 --asr_model ave
python main.py data/May --workspace model/trial_may -O --iters 100000 --finetune_lips --patch_size 64 --asr_model ave

Different ASR models (deepspeech, hubert) and AU45 for eye parameters can be specified.

Train & Test Torso (Repair Double Chin)

To address the "double chin" problem, you can train the torso. Note that the --portrait mode is not supported with torso training.

# Train
python main.py data/May/ --workspace model/trial_may_torso/ -O --torso --head_ckpt model/trial_may/ngp_ep0019.pth --iters 150000 --asr_model ave

# Test
python main.py data/May --workspace model/trial_may_torso -O --torso --test --asr_model ave

Why Use SyncTalk?

SyncTalk stands out for its commitment to high-quality, synchronized talking head synthesis, backed by a CVPR 2024 publication. It offers robust features like preserving subject identity, generating natural facial movements, and restoring fine details. The project provides flexible audio feature encoding options, comprehensive support for both Windows and Linux, and a convenient Google Colab demonstration. Furthermore, its unique torso training capability helps resolve common artifacts such as the "double chin," ensuring more realistic outputs.

Links

• Paper: SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis
• Project Page
• Code Repository
• Colab Notebook Demonstration