Leffa: Controllable Person Image Generation with Flow Fields in Attention

Summary

Leffa is a unified framework for controllable person image generation, enabling precise manipulation of appearance through virtual try-on and pose via pose transfer. This project addresses the common issue of fine-grained textural detail distortion by learning flow fields in attention, guiding target queries to correct reference keys. It achieves state-of-the-art performance, maintaining high image quality while significantly reducing detail distortion.

Use at your own risk

OSRepos shares public repositories for knowledge and discovery only. Any installation, execution, configuration, or use of code from these repositories is the user's own responsibility. Always review the repository, source code, dependencies, licenses, and security implications before running or installing anything. OSRepos is not responsible for issues, damages, or losses resulting from third-party repositories.

Introduction

Leffa (Learning Flow Fields in Attention) is a cutting-edge, unified framework designed for controllable person image generation. Accepted to CVPR 2025, Leffa enables precise manipulation of both appearance, through virtual try-on, and pose, via pose transfer. Traditional methods often struggle with distorting fine-grained textural details from reference images, despite achieving high overall image quality. Leffa tackles this by explicitly guiding the target query to attend to the correct reference key within the attention layer during training, using a regularization loss on top of the attention map. This innovative approach significantly reduces fine-grained detail distortion while maintaining exceptional image quality.

Installation

To get started with Leffa, follow these steps to set up your environment:

conda create -n leffa python==3.10
conda activate leffa
cd Leffa
pip install -r requirements.txt

Examples

Leffa offers robust capabilities for both virtual try-on and pose transfer. The project includes a Gradio application for easy local execution and demonstration. You can also explore the official HuggingFace demo for interactive use. The visualization below showcases Leffa's ability to generate high-quality images with greatly reduced distortion of fine-grained details compared to other methods.

To run the Gradio app locally:

python app.py

Why Use Leffa?

Leffa stands out as a powerful tool for person image generation due to several key advantages:

State-of-the-Art Performance: Achieves superior results in both virtual try-on and pose transfer tasks.
Reduced Detail Distortion: Its unique "flow fields in attention" mechanism effectively preserves fine-grained textural details from reference images.
Unified Framework: Provides a single, cohesive solution for two major controllable person image generation tasks.
Model-Agnostic Loss: The proposed regularization loss can be applied to improve other diffusion models, showcasing its versatility.
Active Development: Regularly updated with performance improvements and new features, as seen in the project's news section.