InfiniteTalk: Unlimited-Length AI Video Generation from Audio or Images

Introduction

InfiniteTalk is a cutting-edge AI model designed for generating unlimited-length talking videos. This powerful framework supports both audio-driven video-to-video and image-to-video generation, offering a versatile solution for creating dynamic visual content. Unlike traditional dubbing methods that primarily focus on lip synchronization, InfiniteTalk synthesizes new videos with accurate lip movements while also aligning head movements, body posture, and facial expressions with the input audio. This ensures a highly realistic and consistent output, making it ideal for various applications from content creation to virtual communication.

Key Features

InfiniteTalk stands out with several key capabilities:

Sparse-frame Video Dubbing: Synchronizes not only lips, but also head, body, and expressions for a natural look.
Infinite-Length Generation: Supports unlimited video duration, overcoming common limitations in AI video generation.
Stability: Reduces hand and body distortions, offering improved visual consistency compared to previous models.
Lip Accuracy: Achieves superior lip synchronization, ensuring that generated speech looks natural and convincing.

Installation

To get started with InfiniteTalk, follow these general steps. For detailed instructions and specific dependencies, please refer to the official GitHub repository.

1. Create a Conda Environment:

conda create -n multitalk python=3.10
conda activate multitalk

2. Install PyTorch and xformers:

pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
pip install -U xformers==0.0.28 --index-url https://download.pytorch.org/whl/cu121

3. Install Flash-attn:

pip install misaki[en] ninja psutil packaging wheel flash_attn==2.7.4.post1

4. Install Other Dependencies:

pip install -r requirements.txt
conda install -c conda-forge librosa

5. FFmpeg Installation:

conda install -c conda-forge ffmpeg

sudo yum install ffmpeg ffmpeg-devel

6. Model Preparation: Download the necessary models (Wan2.1-I2V-14B-480P, chinese-wav2vec2-base, MeiGen-InfiniteTalk) using huggingface-cli as specified in the repository.

Examples

InfiniteTalk provides robust capabilities for both video-to-video and image-to-video generation.

Video-to-Video: Transform existing videos by synchronizing new audio, maintaining the original camera movement and identity. This mode supports unlimited length generation.
Image-to-Video: Generate dynamic talking videos from a single input image and an audio track. This is effective for up to 1 minute, with strategies available for longer high-quality generation.

You can find detailed quick inference commands and various usage scenarios, including single GPU, 720P, low VRAM, multi-GPU, multi-person animation, and integration with FusioniX/Lightx2v, in the official repository. A Gradio demo is also available for easy interaction.

Why Use InfiniteTalk?

InfiniteTalk offers significant advantages for anyone needing advanced audio-driven video generation:

Comprehensive Synchronization: Beyond just lips, it synchronizes head movements, body posture, and facial expressions, leading to more natural and believable results.
Scalability: Its ability to generate videos of unlimited length makes it suitable for long-form content, a major breakthrough in the field.
High Fidelity: The model is designed for stability, reducing common artifacts like hand and body distortions, and achieving superior lip accuracy.
Versatility: Supports both existing video transformation and new video creation from static images, catering to a wide range of creative and practical needs.