SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

Code for "SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis".

Prerequisites

Python 3, NumPy, SciPy, OpenCV 3
Tensorflow(>=1.7.0). Tensorflow 2.0 is not supported.
A recent NVIDIA GPU

Preparations

The path to data files needs to be specified in input_pipeline.py. See below for detailed information on data files.
You need to download "Inception-V4 model", unzip it and put the checkpoint under inception_v4_model.

Dataset

~~Pre-built tfrecord files are available for out of the box training.~~

~~Files for the Sketchy Database can be found here.~~
~~Files for Augmented Sketchy(i.e. flickr images+edge maps), resized to 256x256 regardless of original aspect ratios, can be found here.~~

Note: The webite hosting the dataset is no longer available. Please use the script under data_processing folder to crawl your own images.

If you want to build tfrecord files from images, run flickr_to_tfrecord.py or sketchy_to_tfrecord.py for the respective dataset.

If you wish to get the image files:

The Sketchy Database can be found here.
Use extract_images.py under data_processing to extract images from tfrecord files. You need to specify input and output paths. The extracted images will be sorted by class names.
The dataset I used is no longer availabe due to its large size. You can crawl your own images and run through edge_detection/batch_hed.py -> edge_detection/PostprocessHED.m -> ``flickr_to_tfrecord.py` to create your own dataset.
~~Please contact me if you need the original (not resized) Flickr images, since they are too large to upload to any online space.~~

Configurations

The model can be trained out of the box, by running main_single.py. But there are several places you can change configurations:

Commandline options in main_single.py
Some global options in config.py
Activation/Normalization functions in models_mru.py

Model

The model will be saved periodically. If you wish to resume, just use commandline switch resume_from.
If you wish to test the model, change mode from train to test and fill in resume_from.

Citation

If you use our work for your research, please cite our paper

@InProceedings{Chen_2018_CVPR,
author = {Chen, Wengling and Hays, James},
title = {SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}

Credits

Inception-V4 and VGG16 code by Tensorflow Authors.
Tensorflow implementation of Spectral Normalization by minhnhat93
Improved WGAN

wchen342 / SketchyGAN