dbash / lasagna

Layered Score Distillation for Disentangled Object Relighting

Lasagna: Layered Score Distillation for Disentangled Object Relighting

Dina Bashkirova, Arijit Ray, Rupayan Mallick, Sarah Adel Bargal, Jianming Zhang, Ranjay Krishna, Kate Seanko
arxiv | ReLiT dataset (coming soon)
We propose Lasagna, a layered image editing approach that allows controlled and language-guided object relighting. Lasagna achieves a controlled relighting via layered score distillation sampling that allows extracting the diffusion model lighting prior without changing other crucial aspects of the input image.

Setup

Install dependencies

conda create -y -n lasagna python=3.10.11
conda activate lasagna
conda install -y pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install accelerate
pip install git+https://github.com/huggingface/diffusers

Download the ControlNet checkpoint

wget http://csr.bu.edu/ftp/recycle/lasagna_controlnet.zip
unzip relit_controlnet.zip ./

Training Lasagna to relight an input image

bash train_shading_digital_art.sh