Example-Guided Style-Consistent Image Synthesis from Semantic Labeling

Paper

Example-Guided Style-Consistent Image Synthesis from Semantic Labeling
Miao Wang¹, Guo-Ye Yang², Ruilong Li², Run-Ze Liang², Song-Hai Zhang², Peter M. Hall³ and Shi-Min Hu^2,1
¹State Key Laboratory of Virtual Reality Technology and Systems, Beihang University
²Department of Computer Science and Technology, Tsinghua University, Beijing
³University of Bath
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

Prerequisites

Linux
Python 3
NVIDIA GPU (12G or 24G memory) + CUDA cuDNN
pytorch==0.4.1
numpy
...

Tasks

Sketch2Face

Task name: face

We use the real videos in the FaceForensics dataset, which contains 854 videos of reporters broadcasting news. We localize facial landmarks, crop facial regions and resize them to size 256×256. The detected facial landmarks are connected to create face sketches.

Pose2Dance

Task name: pose

We download 150 solo dance videos from YouTube, crop out the central body regions and resize them to 256×256. We evenly split each video into the ﬁrst part and the second part along the time-line, then sample training data only from the ﬁrst parts and sample testing data only from the second parts of all the videos. The the labels are created using concatenated pre-trained DensePose and OpenPose pose detection results.

SceneParsing2StreetView

Task name: scene

We use the BDD100k dataset to synthesize street view images from pixelwise semantic labels (i.e. scene parsing maps). We use the state-of-the-art scene parsing network DANet to create labels.

Getting Started

Installation

git clone [this project]
cd pix2pixSC
# download datas.zip at https://drive.google.com/drive/folders/1O94UcCXONq7p2ZiPcfi-dldjREQ-GsJK or https://share.weiyun.com/5lHBkE0
unzip datas.zip
mv datas/checkpoints ./
mv datas/datasets ./

# scripts below is optional
mkdir ../FaceForensics
download FaceForensics dataset to ../FaceForensics/datas
python process.py
python generate_data_face_forensics.py --source_path '../FaceForensics/out_data' --target_path './datasets/FaceForensics3/' --same_style_rate 0.3 --neighbor_size 10 --A_repeat_num 50 --copy_data

Training

new_scripts/train_[Task name].sh

Testing

new_scripts/test_[Task name].sh

Inference Face

Edit inference/infer_list.txt, one test case each line, outputs of each test case will be in ./results.

new_scripts/infer_face.sh

Inference code of other tasks will come later.

Results

Face

Dance

Scene

Citation

If you find this useful for your research, please cite the following paper.

@InProceedings{pix2pixSC2019,
author = {Wang, Miao and Yang, Guo-Ye and Li, Ruilong and Liang, Run-Ze and Zhang, Song-Hai and Hall, Peter. M and Hu, Shi-Min},
title = {Example-Guided Style-Consistent Image Synthesis from Semantic Labeling},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}

cxjyxxme / pix2pixSC