GuideWsp / PFNL

Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations

This work is based on Wang et al and Tao et al.

This work has tried to rebuild various state-of-the-art video SR methods, including VESPCN, RVSR-LTD, MCResNet, DRVSR, FRVSR, DUFVSR and PFNL.

Datasets

We have selected MM522 dataset for training and collected another 20 sequences for evaluation, and in consider of copyright, the datasets should only be used for study.

The datasets can be downloaded from Google Drive, train and evaluation.

Note that the training dataset provides Ground Truth images and Bicubic downsampling LR images, while the evaluation dataset provides Gaussian blur and downsampling images. Thus, please refer to ./model/base_model.py for generating Gaussian blur and downsampling images from Ground Truth images.

Unzip the training dataset to ./data/train/ and evaluation dataset to ./data/val/ .

We only provide the ground truth images and the corresponding 4x downsampled LR images by DUFVSR.

Environment

  • Python (Tested on 3.6)
  • Tensorflow (Tested on 1.12.0)

Training

We provide pre-trained models, note that some models have been retrained and part of the codes have been modified, thus some methods may behave a little different from that reported in the paper. Be free to use main.py to train any model you would like to.

Testing

We provide Vid4 and UDM10 testing datasets. It should be easy to use 'testvideo()' or 'testvideos()' functions for testing.

Citation

If you find our code or datasets helpful, please consider citing our related works.

@inproceedings{PFNL,
  title={Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations},
  author={Yi, Peng and Wang, Zhongyuan and Jiang, Kui and Jiang, Junjun and Ma, Jiayi},
  booktitle={IEEE International Conference on Computer Vision (ICCV)},
  pages={3106-3115},
  year={2019},
}

@ARTICLE{wang2018mmcnn,
        author = {Wang, Zhongyuan and Yi, Peng and Jiang, Kui and Jiang, Junjun and Han, Zhen and Lu, Tao and Ma, Jiayi},
        journal={IEEE Transactions on Image Processing},
        title = {Multi-Memory Convolutional Neural Network for Video Super-Resolution},
        year={2018},
    }

@ARTICLE{MTUDM, 
author={Yi, Peng and Wang, Zhongyuan and Jiang, Kui and Shao, Zhenfeng and Ma, Jiayi}, 
journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
title={Multi-Temporal Ultra Dense Memory Network For Video Super-Resolution}, 
year={2019}, 
doi={10.1109/TCSVT.2019.2925844}, 
ISSN={1051-8215},
}

Contact

If you have questions or suggestions, please open an issue here or send an email to yipeng@whu.edu.cn.

Visual Results

We show the visual results under 4x upscaling. This frame is from auditorium in UDM10 testing dataset.

Image text

This frame is from photography in UDM10 testing dataset.

Image text

This is a real LR frame shoot by us.

Image text

PSNR/SSIM on Vid4 test dataset (4xSR)

Sequence VESPCN RVSR-LTD MCResNet DRVSR FRVSR DUF_52L PFNL
calendar 22.20 / 0.7156 22.07 / 0.7041 22.44 / 0.7319 22.88 / 0.7586 23.46 / 0.7854 23.85 / 0.8052 24.37 / 0.8246
city 26.47 / 0.7246 26.44 / 0.7217 26.75 / 0.7454 27.06 / 0.7698 27.70 / 0.8099 27.97 / 0.8253 28.09 / 0.8385
foliage 25.07 / 0.6910 25.15 / 0.7004 25.30 / 0.7093 25.58 / 0.7307 25.96 / 0.7560 26.22 / 0.7646 26.51 / 0.7768
walk 28.40 / 0.8717 28.29 / 0.8677 28.76 / 0.8788 29.11 / 0.8876 29.69 / 0.8990 30.47 / 0.9118 30.64 / 0.9134
average 25.54 / 0.7507 25.49 / 0.7485 25.81 / 0.7664 26.16 / 0.7867 26.70 / 0.8126 27.13 / 0.8267 27.41 / 0.8383
average* 25.35 / 0.7557 - / - 25.45 / 0.7467 25.52 / 0.7600 26.69 / 0.8220 27.34 / 0.8327 27.41 / 0.8383

PSNR/SSIM on UDM10 test dataset (4xSR)

Sequence VESPCN RVSR-LTD MCResNet DRVSR FRVSR DUF_52L PFNL
archpeople 35.37 / 0.9504 35.20 / 0.9485 35.46 / 0.9512 35.83 / 0.9547 36.24 / 0.9579 36.92 / 0.9638 38.35 / 0.9724
archwall 40.14 / 0.9581 39.80 / 0.9559 40.77 / 0.9637 41.16 / 0.9671 41.65 / 0.9710 42.53 / 0.9754 43.55 / 0.9792
auditorium 27.91 / 0.8837 27.49 / 0.8736 27.87 / 0.8874 29.00 / 0.9039 29.81 / 0.9181 30.27 / 0.9257 31.18 / 0.9369
band 33.55 / 0.9514 33.27 / 0.9481 33.88 / 0.9540 34.32 / 0.9579 34.54 / 0.9589 35.49 / 0.9660 36.01 / 0.9691
caffe 37.57 / 0.9647 37.22 / 0.9635 38.07 / 0.9676 39.08 / 0.9715 39.82 / 0.9746 41.03 / 0.9785 41.84 / 0.9808
camera 43.34 / 0.9886 43.36 / 0.9884 43.45 / 0.9887 45.19 / 0.9905 46.07 / 0.9912 47.30 / 0.9927 49.26 / 0.9941
clap 34.92 / 0.9544 34.57 / 0.9511 35.41 / 0.9578 36.20 / 0.9635 36.51 / 0.9659 37.70 / 0.9719 38.33 / 0.9756
lake 30.63 / 0.8255 30.69 / 0.8267 30.82 / 0.8323 31.15 / 0.8440 31.70 / 0.8623 32.06 / 0.8730 32.53 / 0.8865
photography 35.92 / 0.9581 35.61 / 0.9552 36.15 / 0.9594 36.60 / 0.9627 36.95 / 0.9655 38.02 / 0.9719 38.95 / 0.9768
polyflow 36.61 / 0.9489 36.43 / 0.9469 37.01 / 0.9521 37.91 / 0.9565 38.38 / 0.9597 39.25 / 0.9667 40.04 / 0.9734
average 35.60 / 0.9384 35.36 / 0.9358 35.89 / 0.9414 36.64 / 0.9472 37.17 / 0.9525 38.05 / 0.9586 39.00 / 0.9645

About

Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations

License:MIT License


Languages

Language:Python 100.0%