RealSR

Real-World Super-Resolution via Kernel Estimation and Noise Injection (CVPR 2020 Workshops)

Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang

Tencent Youtu Lab

Our solution is the winner of CVPR NTIRE 2020 Challenge on Real-World Super-Resolution in both tracks.

(Official PyTorch Implementation)

Update - September 1, 2020

Release training code at Github/Tencent.

Update - May 26, 2020

Add DF2K-JPEG Model.
Executable files based on ncnn are available. Test your own images on windows/linux/macos. More details refer to realsr-ncnn-vulkan
- Usage - ./realsr-ncnn-vulkan -i in.jpg -o out.png
- -x - use ensemble
- -g 0 - select gpu id.

Introduction

Recent state-of-the-art super-resolution methods have achieved impressive performance on ideal datasets regardless of blur and noise. However, these methods always fail in real-world image super-resolution, since most of them adopt simple bicubic downsampling from high-quality images to construct Low-Resolution (LR) and High-Resolution (HR) pairs for training which may lose track of frequency-related details. To address this issue, we focus on designing a novel degradation framework for real-world images by estimating various blur kernels as well as real noise distributions. Based on our novel degradation framework, we can acquire LR images sharing a common domain with real-world images. Then, we propose a real-world super-resolution model aiming at better perception. Extensive experiments on synthetic noise data and real-world images demonstrate that our method outperforms the state-of-the-art methods, resulting in lower noise and better visual quality. In addition, our method is the winner of NTIRE 2020 Challenge on both tracks of Real-World Super-Resolution, which significantly outperforms other competitors by large margins.

If you are interested in this work, please cite our paper

@InProceedings{Ji_2020_CVPR_Workshops,
               author = {Ji, Xiaozhong and Cao, Yun and Tai, Ying and Wang, Chengjie and Li, Jilin and Huang, Feiyue},
               title = {Real-World Super-Resolution via Kernel Estimation and Noise Injection},
               booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
               month = {June},
               year = {2020}
     }

and challenge report NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results

@article{Lugmayr2020ntire,
        title={NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results},
        author={Andreas Lugmayr, Martin Danelljan, Radu Timofte, Namhyuk Ahn, Dongwoon Bai, Jie Cai, Yun Cao, Junyang Chen, Kaihua Cheng, SeYoung Chun, Wei Deng, Mostafa El-Khamy Chiu, Man Ho, Xiaozhong Ji, Amin Kheradmand, Gwantae Kim, Hanseok Ko, Kanghyu Lee, Jungwon Lee, Hao Li, Ziluan Liu, Zhi-Song Liu, Shuai Liu, Yunhua Lu, Zibo Meng, Pablo Navarrete, Michelini Christian, Micheloni Kalpesh, Prajapati Haoyu, Ren Yong, Hyeok Seo, Wan-Chi Siu, Kyung-Ah Sohn, Ying Tai, Rao Muhammad Umer, Shuangquan Wang, Huibing Wang, Timothy Haoning Wu, Haoning Wu, Biao Yang, Fuzhi Yang, Jaejun Yoo, Tongtong Zhao, Yuanbo Zhou, Haijie Zhuo, Ziyao Zong, Xueyi Zou},
        journal={CVPR Workshops},
        year={2020},
    }

Visual Results

Quantitative Results Compared with Other Participating Methods

'Impressionism' is our team. Note that the final decision is based on MOS (Mean Opinion Score) and MOR (Mean Opinion Rank).

Qualitative Results Compared with Other Participating Methods

'Impressionism' is our team.

Dependencies and Installation

This code is based on BasicSR.

Python 3 (Recommend to use Anaconda)
PyTorch >= 1.0
NVIDIA GPU + CUDA
Python packages: pip install numpy opencv-python lmdb pyyaml
TensorBoard:
- PyTorch >= 1.1: pip install tb-nightly future
- PyTorch == 1.0: pip install tensorboardX

Pre-trained models

Models for challenge results
- DF2K for corrupted images with processing noise.
- DPED for real images taken by cell phone camera.
Extended models
- DF2K-JPEG for compressed jpeg image.

Testing

Download dataset from NTIRE 2020 RWSR and unzip it to your path.

For convenient, we provide Corrupted-te-x and DPEDiphone-crop-te-x.

cd ./codes

DF2K: Image processing artifacts

Modify the configuration file options/df2k/test_df2k.yml
- line 1 : 'name' -- dir name for saving the testing results
- line 13 : 'dataroot_LR' -- test images dir
- line 26 : 'pretrain_model_G' -- pre-trained model for testing
Run command : CUDA_VISIBLE_DEVICES=X python3 test.py -opt options/df2k/test_df2k.yml
The output images is saved in '../results/'

DPED: Smartphone images

Modify the configuration file options/dped/test_dped.yml
- line 1 : 'name' -- dir name for saving the testing results
- line 13 : 'dataroot_LR' -- test images dir
- line 26 : 'pretrain_model_G' -- pre-trained model for testing
Run command : CUDA_VISIBLE_DEVICES=X python3 test.py -opt options/dped/test_dped.yml
The output images is saved in '../results/'

Training

Track 1

prepare training data
- specify dataset paths in './preprocess/path.yml' and create bicubic dataset : python3 ./preprocess/create_bicubic_dataset.py --dataset df2k --artifacts tdsr
- run the below command to collect high frequency noise from Source : python3 ./preprocess/collect_noise.py --dataset df2k --artifacts tdsr
train SR model
- Modify the configuration file options/df2k/train_bicubic_noise.yml
- Run command : CUDA_VISIBLE_DEVICES=4,5,6,7 python3 train.py -opt options/df2k/train_bicubic_noise.yml
- checkpoint dir is in '../experiments'

Track 2

prepare training data
- Use KernelGAN to generate kernels from source images. Clone the repo here. Replace SOURCE_PATH with specific path and run :
```
cd KernelGAN
CUDA_VISIBLE_DEVICES=4,5,6,7 python3 train.py --X4 --input-dir SOURCE_PATH
```
- specify dataset paths in './preprocess/path.yml' and generated KERNEL_PATH to kernel create kernel dataset: python3 ./preprocess/create_kernel_dataset.py --dataset dped --artifacts clean --kernel_path KERNEL_PATH
- run the below command to collect high frequency noise from Source: python3 ./preprocess/collect_noise.py --dataset dped --artifacts clean
train SR model
- Modify the configuration file options/dped/train_kernel_noise.yml
- run command : CUDA_VISIBLE_DEVICES=4,5,6,7 python3 train.py -opt options/dped/train_kernel_noise.yml
- checkpoint dir is in '../experiments'

bigsea00001 / Real-SR