# SyncTalk: High-Quality Talking Head Synthesis from CVPR 2024 This repository profile is provided by osrepos.com, an open source repository discovery platform. Source: osrepos.com Repository profile: https://osrepos.com/repo/ziqiaopeng-synctalk Generated for open source discovery and AI-assisted research. SyncTalk is the official repository for a CVPR 2024 paper on talking head synthesis. This project focuses on generating highly synchronized lip movements, facial expressions, and stable head poses, while also restoring hair details for high-resolution video output. It leverages tri-plane hash representations to maintain subject identity effectively. GitHub: https://github.com/ZiqiaoPeng/SyncTalk OSRepos URL: https://osrepos.com/repo/ziqiaopeng-synctalk ## Summary SyncTalk is the official repository for a CVPR 2024 paper on talking head synthesis. This project focuses on generating highly synchronized lip movements, facial expressions, and stable head poses, while also restoring hair details for high-resolution video output. It leverages tri-plane hash representations to maintain subject identity effectively. ## Topics - Python - Computer Vision - AI - Deep Learning - talking-head - talking-face-generation - cvpr2024 - audio-driven-talking-face ## Repository Information Last analyzed by OSRepos: Thu Dec 11 2025 00:01:06 GMT+0000 (Western European Standard Time) Detail views: 13 GitHub clicks: 12 ## Safety Notice OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything. ## Content ## Introduction SyncTalk is an innovative project presented at CVPR 2024, offering a robust solution for talking head synthesis. It aims to generate highly synchronized talking head videos, ensuring natural lip movements, facial expressions, and stable head poses. The system employs tri-plane hash representations to preserve subject identity and can restore intricate hair details, producing high-resolution video outputs. For even faster and better visual quality, consider exploring the related SyncTalk_2D project. ## Installation SyncTalk supports both Windows and Linux environments. ### For Windows A pre-built integration package is available. Download `SyncTalk-Windows.zip` and unzip it, then double-click `inference.bat` to run the demo. Download links: * Hugging Face * Baidu Netdisk (pwd: 6666) ### For Linux Tested on Ubuntu 18.04, Pytorch 1.12.1 and CUDA 11.3. First, clone the repository: bash git clone https://github.com/ZiqiaoPeng/SyncTalk.git cd SyncTalk Then, set up the environment and install dependencies: bash conda create -n synctalk python==3.8.8 conda activate synctalk pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113 sudo apt-get install portaudio19-dev pip install -r requirements.txt pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1121/download.html pip install tensorflow-gpu==2.8.1 pip install ./freqencoder ./shencoder ./gridencoder ./raymarching If PyTorch3D installation encounters issues, use: bash python ./scripts/install_pytorch3d.py **Data Preparation**: Download pre-trained models (`May.zip` to `data/`, `trial_may.zip` to `model/`) and unzip them. For processing your own video, prepare face-parsing and 3DMM models by running provided `wget` commands and `convert_BFM.py` script. Place your video (`data//.mp4`) and process it: bash python data_utils/process.py data//.mp4 --asr ave Optionally, obtain AU45 for eye blinking using OpenFace. ## Examples ### Run Evaluation Evaluate the model with or without portrait mode: bash python main.py data/May --workspace model/trial_may -O --test --asr_model ave python main.py data/May --workspace model/trial_may -O --test --asr_model ave --portrait ### Inference with Target Audio Generate talking head video from a `.wav` audio file: bash python main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav You can also use DeepSpeech or HuBERT for audio features. ### Training Train a new model or fine-tune an existing one: bash python main.py data/May --workspace model/trial_may -O --iters 60000 --asr_model ave python main.py data/May --workspace model/trial_may -O --iters 100000 --finetune_lips --patch_size 64 --asr_model ave Different ASR models (deepspeech, hubert) and AU45 for eye parameters can be specified. ### Train & Test Torso (Repair Double Chin) To address the "double chin" problem, you can train the torso. Note that the `--portrait` mode is not supported with torso training. bash # Train python main.py data/May/ --workspace model/trial_may_torso/ -O --torso --head_ckpt model/trial_may/ngp_ep0019.pth --iters 150000 --asr_model ave # Test python main.py data/May --workspace model/trial_may_torso -O --torso --test --asr_model ave ## Why Use SyncTalk? SyncTalk stands out for its commitment to high-quality, synchronized talking head synthesis, backed by a CVPR 2024 publication. It offers robust features like preserving subject identity, generating natural facial movements, and restoring fine details. The project provides flexible audio feature encoding options, comprehensive support for both Windows and Linux, and a convenient Google Colab demonstration. Furthermore, its unique torso training capability helps resolve common artifacts such as the "double chin," ensuring more realistic outputs. ## Links * Paper: SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis * Project Page * Code Repository * Colab Notebook Demonstration