LiteFlowNet2

This repository (https://github.com/twhui/LiteFlowNet2) provides the offical release of the code package for my paper A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization (i.e. LiteFlowNet2) in TPAMI 2020 (https://ieeexplore.ieee.org/document/9018073). The arXiv pre-print is available here.

LiteFlowNet2, another lightweight convolutional network, is evolved from our earlier work in CVPR 2018 (https://github.com/twhui/LiteFlowNet) to better address the problem of optical flow estimation by improving flow accuracy and computation time. Comparing to our earlier work, LiteFlowNet2 improves the optical flow accuracy on Sintel clean pass by 23.3%, Sintel final pass by 12.8%, KITTI 2012 by 19.6%, and KITTI 2015 by 18.8%. Its runtime is 2.2 times faster!

	Sintel Clean Testing Set	Sintel Final Testing Set	KITTI12 Testing Set (Out-Noc)	KITTI15 Testing Set (Fl-all)	Model Size (M)	Runtime* (ms) GTX 1080
FlowNet2 (CVPR17)	4.16	5.74	4.82%	10.41%	162	121
PWC-Net+	3.45	4.60	3.36%	7.72%	8.75	40
LiteFlowNet2 (TPAMI)	3.48	4.69	2.63%	7.62%	6.42	40

Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.

LiteFlowNet3

NEW! Our extended work (LiteFlowNet3, ECCV 2020) is now available at https://github.com/twhui/LiteFlowNet3.

We ameliorate the issue of outliers in the cost volume by amending each cost vector through an adaptive modulation prior to the flow decoding. We further improve the flow accuracy by exploring local flow consistency. To this end, each inaccurate optical flow is replaced with an accurate one from a nearby position through a novel warping of the flow field. LiteFlowNet3 not only achieves promising results on public benchmarks but also has a small model size and a fast runtime.

	Sintel Clean Testing Set	Sintel Final Testing Set	KITTI12 Testing Set (Avg-All)	KITTI15 Testing Set (Fl-fg)	Model Size (M)	Runtime* (ms) GTX 1080
LiteFlowNet (CVPR18)	4.54	5.38	1.6	7.99%	5.4	88
LiteFlowNet2 (TPAMI20)	3.48	4.69	1.4	7.64%	6.4	40
HD3 (CVPR19)	4.79	4.67	1.4	9.02%	39.9	128
IRR-PWC (CVPR19)	3.84	4.58	1.6	7.52%	6.4	180
LiteFlowNet3 (ECCV20)	3.03	4.53	1.3	6.96%	5.2	59

Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.

License and Citation

This software and associated documentation files (the "Software"), and the research paper (A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization) including but not limited to the figures, and tables (the "Paper") are provided for academic research purposes only and without any warranty. Any commercial use requires my consent. When using any parts of the Software or the Paper in your work, please cite the following paper:

@InProceedings{hui20liteflownet2,    
 author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
 title = {{A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization}}, 
 journal = {{IEEE Transactions on Pattern Analysis and Machine Intelligence}},
 year = {2020},    
 url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
}

@InProceedings{hui18liteflownet,    
 author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
 title = {{LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation}},    
 booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},    
 year = {2018},    
 pages = {8981--8989},
 url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
}

Prerequisite and Compiling

LiteFlowNet2 uses the same Caffe package as LiteFlowNet. Please refer to the details in LiteFlowNet GitHub repository.

Training

Please refer to the training steps in LiteFlowNet GitHub repository and adopt the training prtocols in LiteFlowNet2 paper.

Trained models

The trained models (LiteFlowNet2-ft-sintel, LiteFlowNet2-ft-kitti) are available in the folder /models/trained. Untar the files to the same folder before you use it.

LiteFlowNet2-ft-sintel: Model used for Sintel benchmark.

LiteFlowNet2-ft-kitti: Model used for KITTI benchmark.

Testing

Open the testing folder

$ cd LiteFlowNet2/models/testing

Create a soft link in the folder /testing

$ ln -s ../../build/tools bin

Replace MODE in ./test_MODE.py to batch if all the images has the same resolution (e.g. Sintel dataset), otherwise replace it to iter (e.g. KITTI dataset).
Replace MODEL in lines 9 and 10 of test_MODE.py to one of the trained models (e.g. LiteFlowNet2-ft-sintel).
Run the testing script. Flow fields (MODEL-0000000.flo, MODEL-0000001.flo, ... etc) are stored in the folder /testing/results having the same order as the image pair sequence.

$ test_MODE.py img1_pathList.txt img2_pathList.txt results

Reimplementation in TensorFlow

A TensorFlow-based reimplementation of LiteFlowNet2 is also available at https://github.com/rogerhcheng/LiteFlowNet2-TF2.

twhui / LiteFlowNet2