Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video
This repository contains the source code for the paper: Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video (ICCV2021). From two consecutive rolling shutter images overlayed on the left, we can generate a high framerate global shutter video on the right.
Installation
Install the dependent packages:
pip install -r requirements.txt
The code is tested with PyTorch 1.6.0 with CUDA 10.2.89.
Note that in our implementation, we borrowed some packages from DeepUnrollNet and SelfDeblur as follows:
Install correlation package
cd ./package_correlation
python setup.py install
Install differentiable forward warping package
cd ./package_forward_warp
python setup.py install
Install core package
cd ./package_core
python setup.py install
Install reblur_package
cd ./reblur_package
python setup.py install
Demo with our pretrained model
Please download the pretrained model and unzip these three subfolders to the deep_unroll_weights
folder of the main directory.
You can now test our model with the provided images in the demo
folder.
To generate the global shutter images corresponding to the middle and first scanlines, simply run
sh demo.sh
To generate multiple global shutter video frames, e.g. 10× temporal upsampling, please run
sh demo_video.sh
The visualization results will be stored in the experiments
folder. Note that additional examples in the dataset can be tested similarly.
Datasets
- Carla-RS and Fastec-RS: Download them to your local computer from here.
Training and evaluating
You can run following commands to re-train the network.
# !! Please update the corresponding paths in 'train_carla.sh' and 'train_fastec.sh' with #
# !! your own local paths, before run following command!! #
sh train_carla.sh
sh train_fastec.sh
You can run following commands to obtain the quantitative evaluations.
# !! Please update the path to test data in 'inference.sh'
# !! with your own local path, before run following command!!
sh inference.sh
Note that --load_1st_GS=0
denotes the correction evaluation corresponding to the middle scanline of second RS frame, and --load_1st_GS=1
denotes the correction evaluation corresponding to the first scanline of second RS frame.
Citations
Please cite our paper if necessary:
@inproceedings{fan_RSSR_ICCV21,
title={Inverting a rolling shutter camera: bring rolling shutter images to high framerate global shutter video},
author={Fan, Bin and Dai, Yuchao},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={4228--4237},
year={2021}
}
@article{fan_RSSR_TPAMI23,
title={Rolling shutter inversion: bring rolling shutter images to high framerate global shutter video},
author={Fan, Bin and Dai, Yuchao and Li, Hongdong},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2023},
volume={45},
number={5},
pages={6214--6230},
publisher={IEEE}
}
Statement
This project is for research purpose only, please contact us for the licence of commercial use. For any other questions or discussion please contact: binfan@mail.nwpu.edu.cn