Code and checkpoints for the paper Finetuning Text-to-Image Diffusion Models for Fairness (ICLR 2024, oral presentation).
Prompt: "A photo of the face of a electrical and electronics repairer, a person". Left images are generated by the original stable-diffusion-v1-5. Right images are generated by the finetuned stable diffusion, debiased for gender and race jointly. For every image, the first color-coded bar denotes the predicted gender: male (blue) or female (red). The second denotes race: WMELH (green), Asian (orange), Black (black), or Indian (brown). WMELH is the broad race class of White, Middle Eastern, and Latino Hispanic. Bar height represents prediction confidence. Bounding boxes denote detected faces. Images with the same number label are generated using the same noise.
This project has been tested under the following environment:
- OS: Red Hat Enterprise Linux 8.4
- GPU: NVIDIA A100-SXM4-40GB (
$\times$ 2, or more) - CUDA Version: 12.2
- We manage environments with Conda. To set up the environment, follow these steps:
conda env create -f environment.yml
conda activate fair-diffusion
- To set up wandb for logging, run:
wandb login
- Download data.zip from this link to the working directory, unzip it by:
unzip data.zip
This will create a data folder in the current working directory.
This repository includes the code and checkpoints necessary to replicate the five experiments detailed in our paper. To find more information on a specific experiment, please click on the experiment's name, which will direct you to the corresponding folder.
| Experiment Name | Prompts | Description |
|---|---|---|
| exp-1-debias-gender | Occupations | Finetune LoRA on text encoder and/or U-Net to debias binary gender to a perfectly balanced distribtion. |
| exp-2-debias-gender-token | Occupations | Finetune prompt prefix to debias binary gender to a perfectly balanced distribtion. |
| exp-3-debias-gender-race | Occupations | Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion. |
| exp-4-debias-gender-race-age | Occupations | Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion. Simultaneouly, control age to 75% yound and 25% old distribution. |
| exp-5-debias-gender-race-multi-concepts | (1) Occupations, (2) occupations with style and context, (3) personal descriptors, and (4) sports | Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion. |
| exp-6-debias-race | Occupations | Finetune LoRA on text encoder and/or U-Net to debias four classes of race, to a perfectly balanced distribtion. |
If you're primarily interested in utilizing and evaluating our debiased Stable Diffusion v1-5, please follow below instructions.
| Experiment Name | Checkpoints | Instructions on how to use |
|---|---|---|
| exp-1-debias-gender | link | unzip to exp-1-debias-gender/outputs, follow instructions starting from this point onwards. |
| exp-2-debias-gender-token | link | unzip to exp-2-debias-gender-token/outputs, follow instructions starting from this point onwards. |
| exp-3-debias-gender-race | link | unzip to exp-3-debias-gender-race/outputs, follow instructions starting from this point onwards. |
| exp-4-debias-gender-race-age | link | unzip to exp-4-debias-gender-race-age/outputs, follow instructions starting from this point onwards. |
| exp-5-debias-gender-race-multi-concepts | link | unzip to exp-5-debias-gender-race-multi-concepts/outputs, follow instructions starting from this point onwards. |
| exp-6-debias-race | link | unzip to exp-6-debias-race/outputs, follow instructions starting from this point onwards. |
We sincerely appreciate your interest in our work. Should you have any questions, please don't hesitate to reach out to Xudong at xudong.shen@u.nus.edu.
@inproceedings{shen2024finetuning,
title={Finetuning Text-to-Image Diffusion Models for Fairness},
author={Xudong Shen and Chao Du and Tianyu Pang and Min Lin and Yongkang Wong and Mohan Kankanhalli},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=hnrB5YHoYu}
}