rohitgandikota / sliders

Concept Sliders for Precise Control of Diffusion Models

Concept Sliders

Project Website | Arxiv Preprint | Trained Sliders | Colab Demo | Huggingface Demo

Official code implementation of "Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models"

Colab Demo

Try out our colab demo here

UPDATE

You can now use GPT-4 (or any other openAI model) to create prompts for your text sliders. All you need to do is describe what slider you want to create (e.g: "i want to make people look happy").
Please refer to the GPT-notebook

Setup

To set up your python environment:

conda create -n sliders python=3.9
conda activate sliders

git  clone https://github.com/rohitgandikota/sliders.git
cd sliders
pip install -r requirements.txt

If you are running on Windows - please refer to these Windows setup guidelines here

Textual Concept Sliders

Training SD-1.x and SD-2.x LoRa

To train an age slider - go to train-scripts/textsliders/data/prompts.yaml and edit the target=person and positive=old person and unconditional=young person (opposite of positive) and neutral=person and action=enhance with guidance=4.
If you do not want your edit to be targetted to person replace it with any target you want (eg. dog) or if you need it global replace person with ""
Finally, run the command:

python trainscripts/textsliders/train_lora.py --attributes 'male, female' --name 'ageslider' --rank 4 --alpha 1 --config_file 'trainscripts/textsliders/data/config.yaml'

--attributes argument is used to disentangle concepts from the slider. For instance age slider makes all old people male (so instead add the "female, male" attributes to allow disentanglement)

Evaluate

To evaluate your trained models use the notebook SD1-sliders-inference.ipynb

Training SD-XL

To train sliders for SD-XL, use the script train_lora_xl.py. The setup is same as SDv1.4

python trainscripts/textsliders/train_lora_xl.py --attributes 'male, female' --name 'agesliderXL' --rank 4 --alpha 1 --config_file 'trainscripts/textsliders/data/config-xl.yaml'

Evaluate

To evaluate your trained models use the notebook XL-sliders-inference.ipynb

Visual Concept Sliders

Training SD-1.x and SD-2.x LoRa

To train image based sliders, you need to create a ~4-6 pairs of image dataset (before/after edit for desired concept). Save the before images and after images separately. You can also create a dataset with varied intensity effect and save them differently.

To train an image slider for eye size - go to train-scripts/imagesliders/data/config.yaml and edit the target=eye and itive='eye' and unconditional='' and neutral=eye and action=enhance with guidance=4.
If you want the diffusion model to figure out the edit concept - leave target, positive, unconditional, neutral as ''
Finally, run the command:

python trainscripts/imagesliders/train_lora-scale.py --name 'eyeslider' --rank 4 --alpha 1 --config_file 'trainscripts/imagesliders/data/config.yaml' --folder_main 'datasets/eyesize/' --folders 'bigsize, smallsize' --scales '1, -1' 

For this to work - you need to store your before images in smallsize and after images in bigsize. The corresponding paired files in both the folders should have same names. Both these subfolders should be under datasets/eyesize. Feel free to make your own datasets in your own named conventions.

Training SD-XL

To train image sliders for SD-XL, use the script train-lora-scale-xl.py. The setup is same as SDv1.4

python trainscripts/imagesliders/train_lora-scale-xl.py --name 'eyesliderXL' --rank 4 --alpha 1 --config_file 'trainscripts/imagesliders/data/config-xl.yaml' --folder_main '/share/u/rohit/imageXLdataset/eyesize_data/'

Editing Real Images

Concept sliders can be used to edit real images. We use null inversion to edit the images - instead of prompt, we use sliders!
Checkout - demo_image_editing.ipynb for mode details.

Running Gradio Demo Locally

You can also run the HF hosted gradio slider tool (huge shoutout to gradio and HF team) locally using the following scripts

git lfs install
git clone https://huggingface.co/spaces/baulab/ConceptSliders
cd ConceptSliders
pip install requirements.txt
python app.py

For more inference time gradio demos please refer to Cameduru's repo here

Citing our work

The preprint can be cited as follows

@article{gandikota2023sliders,
  title={Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models},
  author={Rohit Gandikota and Joanna Materzy\'nska and Tingrui Zhou and Antonio Torralba and David Bau},
  journal={arXiv preprint arXiv:2311.12092},
  year={2023}
}