Homework 1 (Color-Transfer and Texture-Transfer)
We organize this report as follow:
- Training CycleGan
- describe some modification we made (
deconv
vs.upsample + conv
) - shoing the training loss plot
- describe some modification we made (
- Inferencing
- visualize/analysis the difference before/after the modification
- Show some result on personal images
- Comparison with LAB color transfer
1. Training CycleGan
We train CycleGan on summer2winter_yosemite
dataset. The training detailed and modification we made are summarized as follow:
- We use batch size 8 and input resolution
128x128
to fit our time space constraint. - All the other hyperparamters are same as original placeholder.
- We tried original
Deconv
andUpsample + Conv
. Detailed discussion and comparison later. - We use tensorboardX to monitor the training (see below figures)
2. Inferencing
One of the visual defect of original cycle gan is chekerboard artifacts [ref]. The reference propose to use upsample
followed by a conv
to alleviate the artifact. We tried both deconv
and upsample + conv
and showing some of the result as follow:
Input (winter) | deconv CycleGan |
upsample + conv CycleGan |
---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Input (summer) | deconv CycleGan |
upsample + conv CycleGan |
---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
From above figures, we observe the deconv
indeed results more checkerboard artifact than upsample + conv
does. To further prove the idea, we performe a frequency domain analysis by inspecting the spectrum of 2d discrete fourier transform. We take the mean of all spectrum from normal images, images generated from deconv
CycleGan and upsample + conv
CycleGan respectively. Below figures show the mean spectrum of each:
Normal images | deconv CycleGan |
upsample + conv CycleGan |
---|---|---|
![]() |
![]() |
![]() |
The mean spectrum of normal images is smooth with one spotlight in the middle as expected. The spectrum of deconv
have a grid arange spotlights in high frequency area which could cause by the so called checkerboard artifacts. Though there is less checkerboard in upsample + conv
, the spectrum of it have valley in high frequency.
In sum, although the two trained CycleGans are able to transform the style, the artifact are easy to observe if we look detailly.
Below figure showing some result on our captured images by upsample + conv
CycleGan.
Scene | Input | to winter | to summer |
---|---|---|---|
房間窗外 | ![]() |
![]() |
![]() |
沖繩水族館 | ![]() |
![]() |
![]() |
金瓜石 | ![]() |
![]() |
![]() |
[ref]
Comparison with LAB color transferThis is a simple method that shifting the target image LAB according to given image where only mean/std are considered in both side. In our implementation, we randomly sample a source image in the training set to tune given target image color. Below show the results compare with CycleGan:
Input (winter) | upsample + conv CycleGan |
LAB Color Transfer |
---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Input (summer) | upsample + conv CycleGan |
LAB Color Transfer |
---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
The color transfer method perfome well in some of the cases because giving more blue or green could have the people winter/summer feeling. However, if the soure image is not selected carefully, the transfered image will look strange (like the first row in summer2winter table above). Besides, one of the drawback of such color changing method is that it never make new thing. It can't never changing a hourse to zebra. On the other hand, the generative model, here the CycleGan, can make some snow or grass.