(* This research is an extension of Deep Koalarization by Baldassarre et al. Click the link to read the initial research. I also included the original README of the initial research if you want to explore it.)
Colorization of grayscale images brings more life to images and presents a more realistic view of the objects present as in the real world. It can be applied on the restoration of historic monochrome images and films as well as assisting cartoon artists. Previous approaches in image colorization involve scribbling color hints on the input grayscale image or choosing a reference color image to be able to transfer the color traits into the input grayscale image. In this study, the burden of requiring user intervention in the colorization process is removed by using a convolutional neural network(CNN) that is trained on a large dataset of color images. Mode cost is evaluated using L2 regression loss wherein the colorized image is compared with the ground truth image to achieve a relatively close colorization. Previous colorization results show effective colorizations on global image semantics like sky, grass, and trees, but suffers from color bleeding where certain colors exceed their assigned regions. Our proposed approach addresses this problem. We also propose a new approach in colorization using CNN by using ensemble colorization where separate colorizations are performed by two different CNNs then a refinement network combines and refines the results. The effectiveness of the approach is further evaluated by performing a "Colorization Turing Test" wherein participants are tasked to determine which of the given set of images are colorized or real.
(** This part of the README is from the initial research.)
Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2 (2017).
Federico Baldassarre*, Diego Gonzalez Morin* and Lucas Rodés Guirao* (* Authors contributed equally)
This project was developed as part of the DD2424 Deep Learning in Data Science course at KTH Royal Institute of Technology, spring 2017.
The code is built using Keras and Tensorflow.
Consider starring this project if you found it useful ⭐!
If you find Deep Koalarization useful in your research, please consider citing our paper as
@article{deepkoal2017,
author = {Federico Baldassarre, Diego Gonzalez-Morin, Lucas Rodes-Guirao},
title = {Deep-Koalarization: Image Colorization using CNNs and Inception-ResNet-v2},
journal = {ArXiv:1712.03400},
url = {https://arxiv.org/abs/1712.03400},
year = 2017,
month = dec
}
We review some of the most recent approaches to colorize gray-scale images using deep learning methods. Inspired by these, we propose a model which combines a deep Convolutional Neural Network trained from scratch with high-level features extracted from the Inception-ResNet-v2 pre-trained model. Thanks to its fully convolutional architecture, our encoder-decoder model can process images of any size and aspect ratio. Other than presenting the training results, we assess the "public acceptance" of the generated images by means of a user study. Finally, we present a carousel of applications on different types of images, such as historical photographs.
Inspired by Iizuka and Simo-Serra et al. (2016), we combine a deep CNN architecture with Inception-ResNet-v2 pre-trained on ImageNet dataset, which assists the overall colorization process by extracting high-level features. In particular, Inception-ResNet-v2
The fusion between the fixed-size embedding and the intermediary result of the convolutions is performed by means of replication and stacking as described in Iizuka and Simo-Serra et al. (2016).
We have used the MSE loss as the objective function.
The Training data for this experiment could come from any source. We decuded to use ImageNet, which nowadays is considered the de-facto reference for image tasks. This way, it makes easier for others to replicate our experiments.
Refer to INSTRUCTIONS to setup and use the code in this repo.
We are proud if our work gets noticed and helps/inspires other people on their path to knowledge. Here's a list of references we are aware of, some of the authors contacted us, some others we just happened to find online:
- François Chollet tweeted about this project (thank you for Keras)
- Emil Wallnér on FloydHub Blog and freecodecamp
- Amir Kalron on Logz.io Blog
- sparkexpert on CSDN