TD-GEM: Text-Driven Garment Editing Mapper
Reza Dadfar, Sanaz Sabzevari, Mårten Björkman, Danica Kragic
http://arxiv.org/abs/2305.18120Abstract: Language-based fashion image editing allows users to try out variations of desired garments through provided text prompts. Inspired by research on manipulating latent representations in StyleCLIP and HairCLIP, we focus on these latent spaces for editing fashion items of full-body human datasets. Currently, there is a gap in handling fashion image editing due to the complexity of garment shapes and textures and the diversity of human poses. In this paper, we propose an editing optimizer scheme method called TD-GEM, aiming to edit fashion items in a disentangled way. To this end, we initially obtain a latent representation of an image through generative adversarial network inversions such as e4e or PTI for more accurate results. An optimization-based CLIP is then utilized to guide the latent representation of a fashion image in the direction of a target attribute expressed in terms of a text prompt. Our TD-GEM manipulates the image accurately according to the target attribute, while other parts of the image are kept untouched. In the experiments, we evaluate TD-GEM on two different attributes (i.e., “color" and “sleeve length"), which effectively generates realistic images compared to the recent manipulation schemes.
The code is available at github
Go to the Docker folder,
docker build -t dockerUserName/styleclip:latest .
on linux system write
sudo docker run --rm -it -p 8888:8888 --gpus all -v "$(pwd)"/TDGEM:/home/jovyan/work dockerUserName/styleclip:latest /bin/bash
You can drop dockerUserName/ if you create a local image
First, please download the PTI weights: e4e_w+.pt into /pti/.
You can change the following paramters:
/pti/pti_configs/hyperparameters.py:
first_inv_type = 'w+' -> Use pretrained e4e encoder
/pti/pti_configs/paths_config.py:
input_data_path: path of real images
e4e: path of e4e_w+.pt
stylegan2_ada_shhq: pretrained stylegan2-ada model for SHHQ
python run_pti.pyThe following models are required in the "styleGAN-Human/pretrained_models" folder:
- deeplabv3plus-xception-vocNov14_20-51-38_epoch-89.pth
- model_VSECRWKQFQTY_multi_id.pkl
- model_VSECRWKQFQTY_multi_id.pth stylegan_human_v2_1024.pth
Please download them from Pre-trained Model
Go to the folder OP
The opt_clip_delta.py is used to perform image manipulation for an individual file!
python opt_clip_delta.py --image_path "/home/jovyan/work/dataset/ds_200" --base_path "/home/jovyan/work/styleGAN-Human/outputs/ds_200" --results_dir "/home/jovyan/work/results/Op" The run_op_edit.py is a wrapper to run opt_clip_delta.py for a folder of images
python run_op_edit.pyGo to
cd /home/jovyan/work/StyleClip/StyleCLIPThe train and test options are located at /mapper2/options as train_options.py and test_options.py files To run the StyleCLIP please edit the aformentioned files or use the proper arguments. Run the styleCLIP as
python mapper2/scripts/train.pyThe results folder can be given as an arguement --exp_dir path_to_result
The following model are required in the "pretrained_models" folder:
- model_ir_se50.pth
- stylegan2_1024.pth
- stylegan2-ffhq-config-f.pt
- stylegan_human_v2_1024.pth
Please download them from Pre-trained Model
Go to
cd /home/jovyan/work/TDGEM-mainThe train and test options are located at /mapper2/options as train_options.py and test_options.py files To run the TDGEM-main training please edit the aformentioned files or use the proper arguments. Run the TDGEM-main training as
python mapper2/scripts/train.pyand inference as
python mapper2/scripts/inference.pyThe results folder can be given as an arguement --exp_dir path_to_result
The following model are required in the "pretrained_models" folder:
- deeplabv3plus-xception-vocNov14_20-51-38_epoch-89.pth
- parsenet.pth
- stylegan2_1024.pth
Please download them from Pre-trained Model