We present a learning-based approach for removing unwanted obstructions, such as window reflections, fence occlusions or raindrops, from a short sequence of images captured by a moving camera. Our method leverages the motion differences between the background and the obstructing elements to recover both layers. Specifically, we alternate between estimating dense optical flow fields of the two layers and reconstructing each layer from the flowwarped images via a deep convolutional neural network. The learning-based layer reconstruction allows us to accommodate potential errors in the flow estimation and brittle assumptions such as brightness consistency. We show that training on synthetically generated data transfers well to real images. Our results on numerous challenging scenarios of reflection and fence removal demonstrate the effectiveness of the proposed method.
Paper
This is the author's reference implementation of the multi-image reflection removal using TensorFlow described in: "Learning to See Through Obstructions" Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, Jia-Bin Huang (National Taiwan University & Google & Virginia Tech & University of California at Merced & MediaTek Inc.) in CVPR 2020. Should you be making use of our work, please cite our paper [1].
Further information please contact Yu-Lun Liu.
-
- Please overwrite tfoptflow/model_pwcnet.py using the one in this repository.
-
To download the pre-trained models:
- Run your own sequence (reflection removal):
CUDA_VISIBLEDEVICES=0 python3 run_reflection.py- Run your own sequence (fence removal):
CUDA_VISIBLEDEVICES=0 python3 test_fence.py[1] @inproceedings{Liu-Learning-CVPR-2020,
author = {Liu, Yu-Lun and Lai, Wei-Sheng and Yang, Ming-Hsuan and Chuang, Yung-Yu and Huang, Jia-Bin},
title = {Learning to See Through Obstructions},
booktitle = {Conference on Computer Vision and Pattern Recognition},
year = {2020}
}