{"name":"SyncTalk: High-Quality Talking Head Synthesis from CVPR 2024","description":"SyncTalk is the official repository for a CVPR 2024 paper on talking head synthesis. This project focuses on generating highly synchronized lip movements, facial expressions, and stable head poses, while also restoring hair details for high-resolution video output. It leverages tri-plane hash representations to maintain subject identity effectively.","github":"https://github.com/ZiqiaoPeng/SyncTalk","url":"https://osrepos.com/repo/ziqiaopeng-synctalk","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/ziqiaopeng-synctalk","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/ziqiaopeng-synctalk.md","json":"https://osrepos.com/repo/ziqiaopeng-synctalk.json","topics":["Python","Computer Vision","AI","Deep Learning","talking-head","talking-face-generation","cvpr2024","audio-driven-talking-face"],"keywords":["Python","Computer Vision","AI","Deep Learning","talking-head","talking-face-generation","cvpr2024","audio-driven-talking-face"],"stars":null,"summary":"SyncTalk is the official repository for a CVPR 2024 paper on talking head synthesis. This project focuses on generating highly synchronized lip movements, facial expressions, and stable head poses, while also restoring hair details for high-resolution video output. It leverages tri-plane hash representations to maintain subject identity effectively.","content":"## Introduction\nSyncTalk is an innovative project presented at CVPR 2024, offering a robust solution for talking head synthesis. It aims to generate highly synchronized talking head videos, ensuring natural lip movements, facial expressions, and stable head poses. The system employs tri-plane hash representations to preserve subject identity and can restore intricate hair details, producing high-resolution video outputs. For even faster and better visual quality, consider exploring the related <a href=\"https://github.com/ZiqiaoPeng/SyncTalk_2D\" target=\"_blank\">SyncTalk_2D</a> project.\n\n## Installation\nSyncTalk supports both Windows and Linux environments.\n\n### For Windows\nA pre-built integration package is available. Download `SyncTalk-Windows.zip` and unzip it, then double-click `inference.bat` to run the demo.\nDownload links:\n*   <a href=\"https://huggingface.co/ZiqiaoPeng/SyncTalk/blob/main/SyncTalk-Windows.zip\" target=\"_blank\">Hugging Face</a>\n*   <a href=\"https://pan.baidu.com/s/1g3312mZxx__T6rAFPHjrRg?pwd=6666\" target=\"_blank\">Baidu Netdisk (pwd: 6666)</a>\n\n### For Linux\nTested on Ubuntu 18.04, Pytorch 1.12.1 and CUDA 11.3.\n\nFirst, clone the repository:\nbash\ngit clone https://github.com/ZiqiaoPeng/SyncTalk.git\ncd SyncTalk\n\n\nThen, set up the environment and install dependencies:\nbash\nconda create -n synctalk python==3.8.8\nconda activate synctalk\npip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113\nsudo apt-get install portaudio19-dev\npip install -r requirements.txt\npip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1121/download.html\npip install tensorflow-gpu==2.8.1\npip install ./freqencoder ./shencoder ./gridencoder ./raymarching\n\nIf PyTorch3D installation encounters issues, use:\nbash\npython ./scripts/install_pytorch3d.py\n\n\n**Data Preparation**:\nDownload pre-trained models (`May.zip` to `data/`, `trial_may.zip` to `model/`) and unzip them.\nFor processing your own video, prepare face-parsing and 3DMM models by running provided `wget` commands and `convert_BFM.py` script. Place your video (`data/<ID>/<ID>.mp4`) and process it:\nbash\npython data_utils/process.py data/<ID>/<ID>.mp4 --asr ave\n\nOptionally, obtain AU45 for eye blinking using OpenFace.\n\n## Examples\n\n### Run Evaluation\nEvaluate the model with or without portrait mode:\nbash\npython main.py data/May --workspace model/trial_may -O --test --asr_model ave\npython main.py data/May --workspace model/trial_may -O --test --asr_model ave --portrait\n\n\n### Inference with Target Audio\nGenerate talking head video from a `.wav` audio file:\nbash\npython main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav\n\nYou can also use DeepSpeech or HuBERT for audio features.\n\n### Training\nTrain a new model or fine-tune an existing one:\nbash\npython main.py data/May --workspace model/trial_may -O --iters 60000 --asr_model ave\npython main.py data/May --workspace model/trial_may -O --iters 100000 --finetune_lips --patch_size 64 --asr_model ave\n\nDifferent ASR models (deepspeech, hubert) and AU45 for eye parameters can be specified.\n\n### Train & Test Torso (Repair Double Chin)\nTo address the \"double chin\" problem, you can train the torso. Note that the `--portrait` mode is not supported with torso training.\nbash\n# Train\npython main.py data/May/ --workspace model/trial_may_torso/ -O --torso --head_ckpt model/trial_may/ngp_ep0019.pth --iters 150000 --asr_model ave\n\n# Test\npython main.py data/May --workspace model/trial_may_torso -O --torso --test --asr_model ave\n\n\n## Why Use SyncTalk?\nSyncTalk stands out for its commitment to high-quality, synchronized talking head synthesis, backed by a CVPR 2024 publication. It offers robust features like preserving subject identity, generating natural facial movements, and restoring fine details. The project provides flexible audio feature encoding options, comprehensive support for both Windows and Linux, and a convenient Google Colab demonstration. Furthermore, its unique torso training capability helps resolve common artifacts such as the \"double chin,\" ensuring more realistic outputs.\n\n## Links\n*   <a href=\"https://arxiv.org/abs/2311.17590\" target=\"_blank\">Paper: SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis</a>\n*   <a href=\"https://ziqiaopeng.github.io/synctalk/\" target=\"_blank\">Project Page</a>\n*   <a href=\"https://github.com/ZiqiaoPeng/SyncTalk\" target=\"_blank\">Code Repository</a>\n*   <a href=\"https://colab.research.google.com/drive/1Egq0_ZK5sJAAawShxC0y4JRZQuVS2X-Z?usp=sharing\" target=\"_blank\">Colab Notebook Demonstration</a>","metrics":{"detailViews":13,"githubClicks":12},"dates":{"published":null,"modified":"2025-12-11T00:01:06.000Z"}}