Converting Photos to Monet Style Paintings Using cycleGAN And Neural Style Transfer

Problem Definition

Monet paintings are named after French artist Claude Monet, he was one of the most popular Impressionist Era Artist. This project aims to convert normal photos to Monet Style paintings.
I have used two types of deep learning models -

CycleGANs
Neural Style Transfer

Dataset

This is a kaggle competition Dataset. The aim of the competition is to convert photos to monet style paintings.

https://www.kaggle.com/competitions/gan-getting-started/data?select=monet_jpg

There are two types of photos:

Monet Paintings (jpg) - 300 samples
Photo (jpg) - 1000 samples

Data Visualization

Sample Monets

Sample Photos

Visualizing Gray and RGB Channels

Monets

Photos

Data Preprocessing

CycleGAN Train Data - Normalize, Add Random Noise, Random Flip Test Data - Normalize
Neural Style Transfer - resize images (if different shapes)
Adding Random Noise

Random Flip -- Flipping image at random.

Models

CycleGAN - CycleGAN uses a cycle consistency loss to enable training without the need for paired data. In other words, it can translate from one domain to another without a one-to-one mapping between the source and target domain. Image-to-Image Translation of Unpaired Data.
Neural Style Transfer - Style transfer is a computer vision technique that takes two images—a content image and a style reference image—and blends them together so that the resulting output image retains the core elements of the content image, but appears to be “painted” in the style of the style reference image.

Output

Pix2pix

This is a GAN used for paired image-to-image transaltion. This is repurposed for paired image-to-image transalation. The output has lot of noise.

CycleGAN

The cycle Gans has been trained for 10 epochs gave this output. One ecoch training time exceeds 1 hour on google collab TPU. Repurposed pix2pix gave much better output. Two pix2pix model were used (monet-to-photo and image-to-photo).

Neural Style transfer output

The model takes two inputs. The content image is the one we want to transform and the style image is the one whose style we want to capture. Training time is very less, as we use few layers of pretrained VGG19 to extract features from both the images.

FID score

Related file name: calculate_fid.ipynb Sample of 190 images was taken to generate this score. The Frechet Inception Distance score, or FID for short, is a metric that calculates the distance between feature vectors calculated for real and generated images. The FID score is used to evaluate the quality of images generated by generative adversarial networks, and lower scores have been shown to correlate well with higher quality images.

Model	FID score
pix2pix	19,17,342.13
cycleGAN	11,99,486.18
neural style transfer	2,08,174.26

Conclusion

Visual output is much better for Neural style transfer than cycleGAN. Neural style transfer is better for a very specific style but not for scaling output. The cycleGAn is does not produce the best result thus we can try using other types of GAN for this problem.