FlashVideo: Generate and Upscale High-Resolution Videos

Overview

FlashVideo is a research project for generating high-resolution video from text prompts through two stages: generating a 270p video, then enhancing it to 1080p. The approach is intended to make high-resolution generation more efficient by separating initial video creation from detail enhancement.

It is suited to developers and researchers who want to run or study this workflow. The repository provides inference code and weights for both stages, but its README describes training code as incomplete.

Key Features

• Generates 270p video from text prompts with the stage-I model.
• Upscales 270p video to 1080p with the stage-II model.
• Provides inference code and released weights for both stages.
• Supports prompt-file inference across multiple GPUs through a shell script.
• Includes a Jupyter notebook for interactive prompt-based generation.
• Uses a 3D VAE identified in the README as identical to CogVideoX's.

Use Cases

• Researchers can explore a two-stage approach to efficient high-resolution video generation.
• Developers can generate prompt-based videos and then enhance them to 1080p.
• Teams evaluating text-to-video workflows can test inference using the released checkpoints.
• Users with multiple GPUs can submit several prompts from a text file for video generation.

Getting Started

The README tests the project with Python 3.11.11 and PyTorch 2.4.0+cu121. Install dependencies and download the checkpoints:

pip install -r requirements.txt
huggingface-cli download --local-dir ./checkpoints FoundationVision/FlashVideo

See the README for checkpoint setup, notebook use, and prompt-file inference instructions.

Alternatives

• CineScale: CineScale extends pretrained diffusion models to higher-resolution video at inference, rather than using a dedicated 270p-to-1080p pipeline.
• Wan2.2: Wan2.2 offers general text-, image-, and audio-driven video generation, rather than focusing on a two-stage 270p-to-1080p workflow.
• Step-Video-T2V: Step-Video-T2V is a large text-to-video model that generates clips up to 204 frames, rather than a two-stage generation and enhancement system.

Considerations

• The README says both stages were trained with long prompts, and recommends detailed descriptions for best results.
• The default VAE decoder slicing settings are described as suitable for an 80 GB GPU. Lower-memory GPUs may require adjustments to spatial and temporal slicing.
• Training code and dataset augmentation are still listed as work in progress; a Diffusers implementation and Gradio interface are also listed as unavailable.
• Although the repository metadata lists Apache-2.0, the README advises users to consult the original CogVideoX license for usage details.