This repository (https://github.com/twhui/LiteFlowNet2) is the offical release of LiteFlowNet2 for my paper A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization in TPAMI 2020 ( https://ieeexplore.ieee.org/document/9018073). The arXiv pre-print is available here.
LiteFlowNet2, another lightweight convolutional network, is evolved from our earlier work in CVPR 2018 (https://github.com/twhui/LiteFlowNet) to better address the problem of optical flow estimation by improving flow accuracy and computation time. Comparing to our earlier work, LiteFlowNet2 improves the optical flow accuracy on Sintel clean pass by 23.3%, Sintel final pass by 12.8%, KITTI 2012 by 19.6%, and KITTI 2015 by 18.8%. Its runtime is 2.2 times faster!
Sintel Clean Testing Set | Sintel Final Testing Set | KITTI12 Testing Set (Out-Noc) | KITTI15 Testing Set (Fl-all) | Model Size (M) | Runtime* (ms) GTX 1080 | |
---|---|---|---|---|---|---|
FlowNet2 (CVPR17) | 4.16 | 5.74 | 4.82% | 10.41% | 162 | 121 |
PWC-Net+ | 3.45 | 4.60 | 3.36% | 7.72% | 8.75 | 40 |
LiteFlowNet2 (TPAMI) | 3.48 | 4.69 | 2.63% | 7.62% | 6.42 | 40 |
Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.
This software and associated documentation files (the "Software"), and the research paper (A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization) including but not limited to the figures, and tables (the "Paper") are provided for research purposes only and without any warranty. Any commercial use requires my consent. When using any parts of the Software or the Paper in your work, please cite the following paper:
@InProceedings{hui20liteflownet2,
author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},
title = {A {L}ightweight {O}ptical {F}low {CNN} - {R}evisiting {D}ata {F}idelity and {R}egularization},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2020},
url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/}
}
@InProceedings{hui18liteflownet,
author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},
title = {LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation},
booktitle = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2018},
pages = {8981--8989},
url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/}
}
LiteFlowNet2 uses the same Caffe package as LiteFlowNet. Please refer to the details in LiteFlowNet GitHub repository.
Please refer to the training steps in LiteFlowNet GitHub repository and adopt the training prtocols in LiteFlowNet2 paper.
The trained models (LiteFlowNet2-ft-sintel
, LiteFlowNet2-ft-kitti
) are available in the folder /models/trained
. Untar the files to the same folder before you use it.
LiteFlowNet2-ft-sintel
: Model used for Sintel benchmark.
LiteFlowNet2-ft-kitti
: Model used for KITTI benchmark.
- Open the testing folder
$ cd LiteFlowNet2/models/testing
- Create a soft link in the folder
/testing
$ ln -s ../../build/tools bin
-
Replace
MODE
in./test_MODE.py
tobatch
if all the images has the same resolution (e.g. Sintel dataset), otherwise replace it toiter
(e.g. KITTI dataset). -
Replace
MODEL
in line 10 (cnn_model = 'MODEL'
) oftest_MODE.py
to one of the trained models (e.g.LiteFlowNet2-ft-sintel
). -
Run the testing script. Flow fields (
MODEL
-0000000.flo,MODEL
-0000001.flo, ... etc) are stored in the folder/testing/results
having the same order as the image pair sequence.
$ test_MODE.py img1_pathList.txt img2_pathList.txt results
A TensorFlow-based reimplementation of LiteFlowNet2 is also available at https://github.com/rogerhcheng/LiteFlowNet2-TF2.