Syn2Real

Syn2Real Transfer Learning for Image Deraining using Gaussian Processes

Rajeev Yasarla*, Vishwanath A. Sindagi*, Vishal M. Patel

@InProceedings{Yasarla_2020_CVPR,
author = {Yasarla, Rajeev and Sindagi, Vishwanath A. and Patel, Vishal M.},
title = {Syn2Real Transfer Learning for Image Deraining Using Gaussian Processes},
booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

We propose a Gaussian Process-based semi-supervised learning framework which enables the network in learning to derain using synthetic dataset while generalizing better using unlabeled real-world images. Through extensive experiments and ablations on several challenging datasets (such as Rain800, Rain200H and DDN-SIRR), we show that the proposed method, when trained on limited labeled data, achieves on-par performance with fully-labeled training. Additionally, we demonstrate that using unlabeled real-world images in the proposed GP-based framework results in superior performance as compared to existing methods.

Journal extension:

Semi-Supervised Image Deraining using Gaussian Processes

Paper Link

Prerequisites:

Linux
Python 2 or 3
Pytorch version >=1.0
CPU or NVIDIA GPU + CUDA CuDNN (CUDA 8.0)

Dataset structure

download the rain datasets and arrange the rainy images and clean images in the following order
Save the image names into text file (dataset_filename.txt)

   .
    ├── data 
    |   ├── train # Training  
    |   |   ├── derain        
    |   |   |   ├── <dataset_name>   
    |   |   |   |   ├── rain              # rain images 
    |   |   |   |   └── norain            # clean images
    |   |   |   └── dataset_filename.txt
    |   └── test  # Testing
    |   |   ├── derain         
    |   |   |   ├── <dataset_name>          
    |   |   |   |   ├── rain              # rain images 
    |   |   |   |   └── norain            # clean images
    |   |   |   └── dataset_filename.txt

To test Syn2Real:

mention test dataset text file in the line 57 of test.py, for example

    val_filename = 'SIRR_test.txt'

Run the following command

    python test.py -category derain -exp_name DDN_SIRR_withGP

To train Syn2Real:

mention the labeled, unlabeled, and validation dataset in lines 119-121 of train.py, for example

    labeled_name = 'DDN_100_split1.txt'
    unlabeled_name = 'real_input_split1.txt'
    val_filename = 'SIRR_test.txt'

Run the following command to train the base network without Gaussian processes

    python train.py  -train_batch_size 2  -category derain -exp_name DDN_SIRR_withoutGP  -lambda_GP 0.00 -epoch_start 0

Run the following command to train Syn2Real (CVPR'20) model

    python train.py  -train_batch_size 2  -category derain -exp_name DDN_SIRR_withGP  -lambda_GP 0.0015 -epoch_start 0 -version version1

Run the following command to train Syn2Real++ (journal submission GP modellig at feature map level)

    python train.py  -train_batch_size 2  -category derain -exp_name DDN_SIRR_withGP  -lambda_GP 0.0015 -epoch_start 0 -version version2

Cross-domain experiments and Gaussian kernels

cross domain experiments are performed using DIDMDN dataset as source dataset, and other datasets like Rain800, JORDER_200L, DDN as target datasets.

----------------------------------------------------
Source datasets   | Target datasets                | 
----------------------------------------------------
DIDMDN            | Rain800, JORDER_200L, DDN      |
----------------------------------------------------

Gaussian processes can be modelled using different kernels like Linear or Squared_exponential or Rational_quadratic. the updated code provides an option to choose the kernel type

-kernel_type <Linear or Squared_exponential or Rational_quadratic>

hongwang01 / Syn2Real