Leffa: Generate Controllable Person Images

Overview

Leffa is a framework for generating person images with control over clothing appearance or pose. It addresses a common issue in these tasks: generated images can look convincing overall while losing fine details from the reference image.

Its approach adds a regularization loss to attention maps during training, guiding target regions toward corresponding reference regions. The project includes released models and a Hugging Face demo, and is based on a paper accepted to CVPR 2025. See the paper for the method and evaluation details.

Key Features

• Supports virtual try-on for changing a person's clothing based on a reference garment.
• Supports pose transfer to generate a person image in a target pose.
• Uses attention-map regularization to guide correspondence between reference and generated regions.
• Applies the loss on top of a diffusion-based baseline and is described as model-agnostic.
• Provides released virtual try-on and pose-transfer models through the Hugging Face model page.
• Includes a Gradio app that can be run locally.

Use Cases

• Researchers comparing methods for virtual try-on or pose transfer can use the released models and the paper's evaluation results as a starting point.
• Computer-vision developers can explore attention-guided training to reduce loss of garment or appearance details in generated images.
• Teams prototyping person-image editing can try the hosted demo before setting up the local application.
• Developers working with diffusion models can investigate whether the attention regularization approach transfers to another model.

Getting Started

The README recommends Python 3.10 and a conda environment:

conda create -n leffa python==3.10
conda activate leffa
cd Leffa
pip install -r requirements.txt
python app.py

See the README for further setup and usage details.

Considerations

• The repository provides a training method and inference application, so it is more suited to developers and researchers than to users seeking a standalone image editor.
• The README reports an inference time of 6 seconds on an A100 with float16 defaults. This is a hardware-specific report, not a general performance guarantee.
• The demo uses SCHP and DensePose for mask and pose-related inputs. Local setup and model use may require additional dependencies and compute resources.
• The README describes released models trained on VITON-HD, DressCode, and DeepFashion; results and applicability may depend on the task and data.