facebookresearch

audio2photoreal: Generate Photorealistic Avatars from Audio

A PyTorch research project for generating photorealistic human face and body motion from conversational audio. It includes person-specific pretrained models, training code, and access to the annotated dataset.

View on GitHub Compare
Stars
2.8k
Forks
283
Stars, last 30 days
-1

Our take

Health score

28/100

How it is scored
Activity
19
Community
51
Issues
21
Pull requests
20

-1 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

2,846

Forks

282

Change since Nov 15, 2026: 0 stars·0 forks

Overview

audio2photoreal is a PyTorch implementation of research that synthesizes photorealistic human avatars for conversation from audio. It generates facial expression and body motion, then uses a separate rendering pipeline to produce avatar video.

The project is aimed at researchers and developers exploring audio-driven avatar animation, conversational embodiment, or reproduction of the paper's results. It provides code, pretrained models, and dataset access, but generating rendered video also depends on the Codec Avatar Body pipeline.

Key Features

  • Generates facial expression codes conditioned on audio using a diffusion model.
  • Generates body pose using audio, guide poses, and a body diffusion model.
  • Provides pretrained, person-specific motion models for four dataset participants.
  • Includes training scripts for face diffusion, body diffusion, a body VQ-VAE, and a body guide transformer.
  • Provides annotated audio, body-pose, and face-expression data, with scripts to download and visualize it.
  • Offers a Gradio demo for recording audio and generating avatar video.
  • Integrates with the Codec Avatar Body renderer for photorealistic visualization.

Use Cases

  • Research replication: computer-vision and machine-learning researchers can reproduce or extend the paper's audio-driven avatar experiments.
  • Conversational avatar prototyping: developers can test generated face and body motion from recorded conversational audio.
  • Motion-generation research: teams can train or evaluate person-specific models using the provided annotations and training scripts.
  • Dataset inspection: researchers can render ground-truth annotations to examine audio, pose, and facial-code sequences.

Getting Started

The README documents a CUDA 11.7 and Python 3.9 setup. For the demo, create the environment and run the installer:

conda create --name a2p_env python=3.9
conda activate a2p_env
sh demo/install.sh
python -m demo.demo

See the README for model and dataset downloads, training commands, and rendering instructions.

Alternatives

  • HunyuanVideo-Avatar: Generates emotion-controllable character videos from avatar images and audio, rather than focusing on person-specific face and body motion models.
  • SyncTalk: Creates audio-synchronized talking-head videos from a subject’s footage, with less emphasis on full-body motion.
  • InfiniteTalk: Animates an image or existing video from audio, including facial expressions and body movement, rather than centering on person-specific pretrained models.
ProjectLanguageLicenseStarsStatus
audio2photorealPythonOther2.8kNot checked yet
HunyuanVideo-AvatarPythonOther2.2kQuiet
SyncTalkPythonOther1.6kInactive
InfiniteTalkPythonApache-2.08kQuiet

Considerations

  • The repository is archived, so ongoing maintenance should not be expected.
  • The demo and rendering setup is specialized: the README specifies CUDA 11.7, Python 3.9, GCC 9.0, and PyTorch3D compatibility, and points to a separate renderer repository.
  • Models are person-specific, and the workflow requires downloading the relevant datasets and model assets.
  • The repository states that code and dataset are released under CC-NC 4.0. Check the license file before using them, especially for commercial purposes.

Found this useful?

Share it with someone who would like audio2photoreal.

Comparisons