RayMusk / fast_neural_style_transfer

Generative Neural Methods Based On Model Iteration

Fast-Neural-Style-Transfer

Generative Neural Methods Based On Model Iteration

Description

Using a Deep Residual Convolutional Neural Network as an Image Transformation Network (ITN). We train the ITN to transform input images into output images. We use a VGG19 which is pre-trained on ImageNet dataset as Loss Network to define two Perceptual Losses (Feature Reconstruction Loss & Style Reconstruction Loss) that measure perceptual differences in content and style between images.
For a specific style, we train one ITN using MS-COCO dataset (about 12.6GB). After trained, we can use the ITN to transfer specific style to any image just performing one feed-forward computation.

This code is based on Johnson et al. Perceptual Losses for Real-Time Style Transfer and Super-Resolution [2016.03] and Ulyanov et al. Instance Normalization: The Missing Ingredient for Fast Stylization [2016.09].

System overview. Picture comes from Johnson et al. original paper. In my implementation, I replace the VGG-16 to a pre-trained VGG-19. I use 'relu4_2' layer to compute Feature Reconstruction Loss and use {'relu1_1', 'relu2_1', 'relu3_1', 'relu4_1', 'relu5_1'} layers to compute Style Reconstruction Loss.

Results

style	output (generated image)

Manual

• The main file main.py is a demo, which has already contained training procedure and inferring procedure (inferring means generating stylized images).
  You can switch these two procedures by changing the flag IS_TRAINING.
• By default,
  (1) The content images lie in the folder "./images/content/"
  (2) The style images lie in the folder "./images/style/"
  (3) The weights file of the pre-trained VGG-19 lies in the current working directory. (See Prerequisites below)
  (4) The MS-COCO images dataset for training lies in the folder "./MS_COCO/" (See Prerequisites below)
  (5) The checkpoint files of trained models lie in the folder "./models/" (You should create this folder manually before training)
  (6) After inferring procedure, the stylized images will be generated and put in the folder "./outputs/"
• For training, you should make sure (2), (3), (4) and (5) are prepared correctly.
• For inferring, you should make sure (1) and (5) are prepared correctly.
• Of course, you can organize all the files and folders as you want, and what you need to do is just modifying the main.py file.

Prerequisites

• Pre-trained VGG19 network (MD5 8ee3263992981a1d26e73b3ca028a123)
  I have provided a convertor in the tool folder. It can convert the matlab file into a npz file which is much smaller and easier to process via NumPy.
  Or simply download my pre-processed Pre-trained VGG19 network npz format (MD5 c7ddd13b12e40033b5031ff43e467065) The npz format is about 80MB while the mat format is about 550MB.
  Chinese user can also use BaiduYun to download from here.
• Microsoft COCO dataset

Trained Models

I have trained models over 10 styles: wave, udnie, escher_sphere, flower, scream, denoised_starry, starry_bright, rain_princess, woman_matisse, mosaic. (You can click the "models" to download)
Chinese user can also use BaiduYun to download from here.

My Running Environment

Hardware

• CPU: Intel® Core™ i9-7900X (3.30GHz x 10 cores, 20 threads)
• GPU: NVIDIA® Titan Xp (Architecture: Pascal, Frame buffer: 12GB)
• Memory: 32GB DDR4

Operating System

• ubuntu 16.04.03 LTS

Software

• Python 3.6.2
• NumPy 1.13.1
• TensorFlow 1.3.0
• SciPy 0.19.1
• CUDA 8.0.61
• cuDNN 6.0.21

References

• This project borrowed some ideas and paradigms from Logan Engstrom's Fast Style Transfer and Zhiyuan He's Fast Neural Style Tensorflow
• Parts of README formatting were copied from Justin Johnson's Fast Neural Style and Zhiyuan He's Fast Neural Style Tensorflow

Citation

  @misc{ye2017fastneuralstyletransfer,
    author = {Wengao Ye},
    title = {Fast Neural Style Transfer},
    year = {2017},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/elleryqueenhomels/fast-neural-style-transfer}}
  }