This repository contains python scripts for training and testing Deep Global Registration, CVPR 2020 Oral. Deep Global Registration (DGR) proposes a differentiable framework for pairwise registration of real-world 3D scans. DGR consists of the following three modules:
- a 6-dimensional convolutional network for correspondence confidence prediction
- a differentiable Weighted Procrustes algorithm for closed-form pose estimation
- a robust gradient-based SE(3) optimizer for pose refinement.
For more details, please check out
| Indoor 3DMatch Registration | Outdoor KITTI Lidar Registration |
|---|---|
 |
 |
Recent end-to-end frameworks combine feature learning and pose optimization. PointNetLK combines PointNet global features with an iterative pose optimization method. Wang et al. in Deep Closest Point train graph neural network features by backpropagating through pose optimization. We further advance this line of work. In particular, our Weighted Procrustes method reduces the complexity of optimization from quadratic to linear and enables the use of dense correspondences for highly accurate registration of real-world scans.
The first component is a 6-dimensional convolutional network that analyzes the geometry of 3D correspondences and estimates their accuracy. Please refer to High-dim ConvNets, CVPR'20 for more details.
The second component we develop is a differentiable Weighted Procrustes solver. The Procrustes method provides a closed-form solution for rigid registration in SE(3). A differentiable version of the Procrustes method used for end-to-end registration passes gradients through coordinates, which requires O(N^2) time and memory for N keypoints. Instead, the Weighted Procrustes method passes gradients through the weights associated with correspondences rather than correspondence coordinates. The computational complexity of the Weighted Procrustes method is linear to the number of correspondences, allowing the registration pipeline to use dense correspondence sets rather than sparse keypoints. This substantially increases registration accuracy.
Our third component is a robust optimization module that fine-tunes the alignment produced by the Weighted Procrustes solver and the failure detection module. This optimization module minimizes a differentiable loss via gradient descent on the continuous SE(3) representation space. The optimization is fast since it does not require neighbor search in the inner loop such as ICP.
Our network is built on the MinkowskiEngine and the system requirements are:
- Ubuntu 14.04 or higher
- CUDA 10.1.243 or higher
- pytorch 1.5 or higher
- python 3.6 or higher
- GCC 7
You can install the MinkowskiEngine and the python requirements on your system with:
# Install MinkowskiEngine
sudo apt install libopenblas-dev g++-7
pip install torch
export CXX=g++-7; pip install -U MinkowskiEngine --install-option="--blas=openblas" -v
# Download and setup DeepGlobalRegistration
git clone https://github.com/chrischoy/DeepGlobalRegistration.git
cd DeepGlobalRegistration
pip install -r requirements.txtYou may register your own data with relevant pretrained DGR models. 3DMatch is suitable for indoor RGB-D scans; KITTI is for outdoor LiDAR scans.
| Inlier Model | FCGF model | Dataset | Voxel Size | Feature Dimension | Performance | Link |
|---|---|---|---|---|---|---|
| ResUNetBN2C | ResUNetBN2C | 3DMatch | 5cm (0.05) | 32 | TE: 7.34cm, RE: 2.43deg | weights |
| ResUNetBN2C | ResUNetBN2C | KITTI | 30cm (0.3) | 32 | TE: 3.14cm, RE: 0.14deg | weights |
python demo.py| Input PointClouds | Output Prediction |
|---|---|
 |
 |
| Comparison | Speed vs. Recall Pareto Frontier |
|---|---|
 |
 |
The entire network depends on pretrained FCGF models. Please download corresponding models before training.
| Model | Normalized Feature | Dataset | Voxel Size | Feature Dimension | Link |
|---|---|---|---|---|---|
| ResUNetBN2C | True | 3DMatch | 5cm (0.05) | 32 | download |
| ResUNetBN2C | True | KITTI | 30cm (0.3) | 32 | download |
You may download preprocessed data and train via these commands:
./scripts/download_3dmatch.sh /path/to/3dmatch
export THREED_MATCH_DIR=/path/to/3dmatch; FCGF_WEIGHTS=/path/to/fcgf_3dmatch.pth ./scripts/train_3dmatch.shFollow the instruction on KITTI Odometry website to download the KITTI odometry train set. Then train with
export KITTI_PATH=/path/to/kitti; FCGF_WEIGHTS=/path/to/fcgf_kitti.pth ./scripts/train_kitti.sh3DMatch test set is different from train set and is available at the download section of the official website. You may download and decompress these scenes to a new folder.
To evaluate trained model on 3DMatch or KITTI, you may use
python -m scripts.test_3dmatch --threed_match_dir /path/to/3dmatch_test/ --weights /path/to/dgr_3dmatch.pthand
python -m scripts.test_kitti --kitti_dir /path/to/kitti/ --weights /path/to/dgr_kitti.pthWe also provide experimental results of 3DMatch comparisons in results.npz. To reproduce figures we presented in the paper, you may use
python scripts/analyze_stats.py assets/results.npzPlease cite the following papers if you use our code:
@inproceedings{choy2020deep,
title={Deep Global Registration},
author={Choy, Christopher and Dong, Wei and Koltun, Vladlen},
booktitle={CVPR},
year={2020}
}
@inproceedings{choy2019fully,
title = {Fully Convolutional Geometric Features},
author = {Choy, Christopher and Park, Jaesik and Koltun, Vladlen},
booktitle = {ICCV},
year = {2019}
}
@inproceedings{choy20194d,
title={4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks},
author={Choy, Christopher and Gwak, JunYoung and Savarese, Silvio},
booktitle={CVPR},
year={2019}
}There have a number of 3D registration works published concurrently.