Deep Koalarization

Image colorization using deep CNN and features extracted from Inception-ResNet-v2

This project was developed as part of the DD2424 Deep Learning in Data Science course at KTH Royal Institute of Technology, spring 2017.

We attach the full report and our slides.

Author	GitHub
Federico Baldassarre	baldassarreFe
Diego Gonzalez Morin	diegomorin8
Lucas Rodés Guirao	lucasrodes

Please star our work if you find it useful ⭐!

Intro

We got the inspiration from the work of Richard Zhang, Phillip Isola and Alexei A. Efros, who realized a network able to colorize black and white images (blog post and paper). They trained a network on ImageNet pictures preprocessed to make them gray-scale, with the colored image as the output target.

Then we also saw the experiments of Satoshi Iizuka, Edgar Simo-Serra and Hiroshi Ishikawa, who added image classification features to raw pixels fed to the network, improving the overall results (YouTube review, blog post and paper).

Network

In our work we implement a similar architecture, enriching the pixel information of each image with semantic information about its content. However, object recognition is already a complex problem and would be worth a project on its own. For this reason we are not going to build an image classifier from scratch. Instead we will use a network that has already been trained on a large variety of images, such as Google’s Inception-ResNet-v2, which is trained on ImageNet.

The hidden layers of these models learned to create a semantic representation of the image that is then used by the final layer (fully connected + softmax) to label the objects in the image. By “cutting” the model at one of its final layers we will get a high dimensional representation of image features, that will be used by our network to perform the colorization task (TensorFlow tutorial on transfer learning, another tutorial and arXiv paper).

The fusion between the fixed-size embedding and the intermediary result of the convolutions is performed by means of replication and stacking as described in the original paper:

We have used the MSE loss as the objective function.

Dataset

Training data for this experiment could actually come from every source, but we rather prefer using images from ImageNet, which nowadays is considered the de-facto reference for image tasks. It will also make our experiments easily reproducible, compared to using our own images.

Software packages and instructions

Our project is implemented in Python 3.6, for it is widely used in the data science and machine learning communities. As for additional libraries, for machine learning, deep learning and image manipulation we rely on Keras/TensorFlow, NumPy/SciPy, Scikit.

Refer to INSTRUCTIONS to setup and use the code in this repo.

Results

ImageNet

Historical pictures

Thanks to the people who noticed our work!

We are proud if our work gets noticed and helps/inspires other people on their path to knowledge. Here's a list of references we are aware of, some of the authors contacted us, some others we just happened to find online:

François Chollet tweeted about this project (thank you for Keras)
Emil Wallnér on FloydHub Blog
Amir Kalron on Logz.io Blog
sparkexpert on CSDN

Citation

Please cite this GitHub repo as

@misc{deepkoal2017,
  author = {Federico Baldassarre, Diego Gonzalez-Morin, Lucas Rodes-Guirao},
  title = {Deep-Koalarization},
  year = {2017},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/baldassarreFe/deep-koalarization}},
}

or our report as

@misc{deepkoal2017,
  author = {Federico Baldassarre, Diego Gonzalez-Morin, Lucas Rodes-Guirao},
  title = {Deep-Koalarization: Image Colorization using CNNs and Inception-Resnet-v2},
  year = {2017},
  howpublished = {\url{https://github.com/baldassarreFe/deep-koalarization/blob/master/report.pdf}},
}

nanfengpo / deep-koalarization