This is the implementation of paper "Expanding Sparse LiDAR Depth and Guiding Stereo Matching for Robust Dense Depth Estimation" on RAL 2023.
Camera ready version and meterials will be ginven.
Dense depth estimation is an important task for various applications, such as object detection, 3-D reconstruction, etc. Stereo matching, as a popular method for dense depth estimation, is faced with challenges when low textures, occlusions or domain gaps exist. Stereo-LiDAR fusion has recently become a promising way to deal with these challenges. However, due to the sparsity and uneven distribution of the LiDAR depth data, existing stereo-LiDAR fusion methods tend to ignore the data when their density is quite low or they largely differ from the depth predicted from stereo images. To provide a solution to this problem, we propose a stereo-LiDAR fusion method by first expanding the sparse LiDAR depth to semi-dense depth with RGB image as reference. Based on the semi-dense depth, a varying-weight Gaussian guiding method is proposed to deal with the varying reliability of guiding signals. A multi-scale feature extraction and fusion method is further used to enhance the network, which shows superior performance than traditional sparse invariant convolution methods. Experimental results on different public datasets demonstrate our superior accuracy and robustness over the state-of-the-arts.
The demo for this work is given here, where CE-NET, GSM is chosen for comparison.
- python 3.6.2
- pytorch 1.10.1
- numpy 1.19.2
Download Scene Flow Datasets, KITTI 2012, KITTI 2015, KITTI COMPLETION, Middlebury
KITTI2015/2012 SceneFlow
please place the dataset as described in "./filenames", i.e., "./filenames/sceneflow_train.txt", "./filenames/sceneflow_test.txt", "./filenames/kitti_completion_train.txt", "./filenames/kitti_completion_test.txt"
Scene Flow Datasets Pretraining
run the script ./scripts/sceneflow.sh to pre-train on Scene Flow datsets. Please update DATAPATH in the bash file as your training data path.
To repeat our pretraining details.
Kitti completion Datasets Pretraining
run the script ./scripts/kitti_completion.sh to pre-train on KITTI completion datsets. Please update DATAPATH in the bash file as your training data path.
To repeat our pretraining details.
run the script ./scripts/kitti_completion_test.sh to evaluate the performance on KITTI completion datsets.
Pretraining Model You can use this checkpoint to reproduce the result we reported in Table.I of the main paper.
If you find this code useful in your research, please cite:
@ARTICLE{10026457,
author={Xu, Zhenyu and Li, Yuehua and Zhu, Shiqiang and Sun, Yuxiang},
journal={IEEE Robotics and Automation Letters},
title={Expanding Sparse LiDAR Depth and Guiding Stereo Matching for Robust Dense Depth Estimation},
year={2023},
volume={8},
number={3},
pages={1479-1486},
doi={10.1109/LRA.2023.3240093}}
Thanks to the excellent work CF-NET, GWCNet, Deeppruner, and HSMNet. Our work is inspired by these work and part of codes are migrated from CF-NET, GWCNet, DeepPruner and HSMNet.