HpWang-whu / RoReg

[TPAMI 2023] RoReg: Pairwise Point Cloud Registration with Oriented Descriptors and Local Rotations

🤗 RoReg

RoReg: Pairwise Point Cloud Registration with Oriented Descriptors and Local Rotations

IEEE TPAMI 2023

Haiping Wang^*,1, Yuan Liu^*,2, Qingyong Hu³, Bing Wang⁴, Jianguo Chen⁵, Zhen Dong^†,1, Yulan Guo⁶, Wenping Wang⁷ Bisheng Yang^†,1

¹Wuhan University    ²The University of Hong Kong    ³University of Oxford   
⁴The Hong Kong Polytechnic University    ⁵DiDi Chuxing   
⁶Sun Yat-sen University    ⁷Texas A&M University   
^*The first two authors contribute equally.    ^†Corresponding authors.   

We present RoReg, a novel point cloud registration framework that fully exploits oriented descriptors and estimated local rotations in the whole registration pipeline. Previous methods mainly focus on extracting rotation-invariant descriptors for registration but unanimously neglect the orientations of descriptors. In this paper, we show that the oriented descriptors and the estimated local rotations are very useful in the whole registration pipeline, including feature description, feature detection, feature matching, and transformation estimation. Consequently, we design a novel oriented descriptor RoReg-Desc and apply RoReg-Desc to estimate the local rotations. Such estimated local rotations enable us to develop a rotation-guided detector, a rotation coherence matcher, and a one-shot-estimation RANSAC, all of which greatly improve the registration performance. Extensive experiments demonstrate that RoReg achieves state-of-the-art performance on the widely-used 3DMatch and 3DLoMatch datasets, and also generalizes well to the outdoor ETH dataset. In particular, we also provide in-depth analysis on each component of RoReg, validating the improvements brought by oriented descriptors and the estimated local rotations.

🆕 News

• 2023-02-28: A multiview registration mehtod SGHR is accepted by CVPR 2023! 🎉🎉

• 2023-02-05 RoReg has been accepted by IEEE TPAMI! 🎉🎉

  [TPAMI]

  [Project_Page]

  [Supplementary_Material]

• 2022-09-16 The code of RoReg is released.

• 2022-06-30 The early work of RoReg, a.k.a YOHO, is accepted by ACM MM 2022! 🎉🎉

💻 Requirements

Here we offer the FCGF backbone RoReg. Thus FCGF requirements need to be met:

• Ubuntu 14.04 or higher
• CUDA 11.1 or higher
• Python v3.7 or higher
• Pytorch v1.6 or higher

Specifically, The code has been tested with:

• Ubuntu 16.04, CUDA 11.1, python 3.7.10, Pytorch 1.7.1, GeForce RTX 2080Ti.

🔧 Installation

• First, create the conda environment:

    conda create -n roreg python=3.7
    conda activate roreg
  

• Second, intall Pytorch. We have checked version 1.7.1 and other versions can be referred to Official Set.

    conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
  

• Third, install MinkowskiEngine v0.5 for FCGF feature extraction:

    cd utils/MinkowskiEngine
    conda install openblas-devel -c anaconda
    export CUDA_HOME=/usr/local/cuda-11.1 #We have checked cuda-11.1.
    python setup.py install --blas_include_dirs=${CONDA_PREFIX}/include --blas=openblas
    cd ../..
  

• Fourth, install other packages, here we use 0.8.0.0 version Open3d for Ubuntu 16.04:

  pip install -r requirements.txt
  

💾 Dataset & Pretrained model

The datasets and pretrained weights have been uploaded to Google Cloud:

• 3dm_train_rot;
• 3DMatch/3DLomatch;
• ETH;
• Pretrained Weights. (Already added to the main branch.)

Also, all data above can be downloaded in BaiduDisk(Code:b4zd), where the checkpoints of RoReg and 3dm_train_rot are saved in YOHO/RoReg.

Datasets above contain the point clouds (.ply) and keypoints (.txt, 5000 per point cloud) files. Please place the data to ./data/origin_data following the example data structure as:

data/
├── origin_data/
    ├── 3dmatch/
    	└── kitchen/
            ├── PointCloud/
            	├── cloud_bin_0.ply
            	├── gt.log
            	└── gt.info
            └── Keypoints/
            	└── cloud_bin_0Keypoints.txt
    ├── 3dmatch_train/
    └── ETH/

🚅 Train

To train RoReg-Desc and local rotation estimation (one-shot transformation estimation) with the FCGF backbone we offer, you can first prepare the trainset:

python trainset.py --component GF

and conduct training of the two components by:

python Train.py --component GF # for RoReg-Desc, requiring ~250G storage space.
python Train.py --component ET # for local rotations

After the training of RoReg-Desc and local rotation estimation, you can follow the commonds to train rotation-guided detector yourself:

python trainset.py --component RD
python Train.py --component RD

To train rotation coherence matcher yourself, you can follow the commonds of

python trainset.py --component RM # require ~300G storage space.
python Train.py --component RM

All models will be saved in ./checkpoints/FCGF.

🔦 Demo

With the pretrained models, you can try RoReg with:

python demo.py

✏️ Test on the 3DMatch and 3DLoMatch

With the TestData downloaded above, the test on 3DMatch and 3DLoMatch can be done by

• Prepare the testset

python testset.py --dataset 3dmatch --voxel_size 0.025

• Eval the results:

python Test.py --RD --RM --ET yohoo --keynum 1000 --testset 3dmatch

--RD denotes using the proposed rotation-guided detector, and we will use randomly-sampling without it.

--RM denotes using the proposed rotation coherence matcher, and we will use NN+mutual without it.

--ET denotes the choice of transformation estimation. yohoo means using the proposed one-shot transformation estimation, and we also offer another faster RANSAC variant yohoc.

--keynum denotes to sample how many keypoints in each scan for matching.

--dataset denotes the evaluation dataset: 3dmatch for the 3DMatch dataset and 3dLomatch for the 3DLoMatch dataset.

More options as well as their descriptions can be found in Test.py.

All the results will be placed to ./data/YOHO_FCGF.

✒️ Generalize to the ETH dataset

The generalization results on the outdoor ETH dataset can be got as follows:

• Prepare the testset

python testset.py --dataset ETH --voxel_size 0.15

• Eval the results:

python Test.py --RD --RM --ET yohoo --keynum 1000 --testset ETH --tau_2 0.2 --tau_3 0.5 --ransac_ird 0.5

All the results will be placed to ./data/YOHO_FCGF.

💡 Citation

If you find RoReg/YOHO useful in your research, please consider citing:

@inproceedings{wang2022you,
  title={You only hypothesize once: Point cloud registration with rotation-equivariant descriptors},
  author={Wang, Haiping and Liu, Yuan and Dong, Zhen and Wang, Wenping},
  booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
  pages={1630--1641},
  year={2022}

@ARTICLE{wang2023roreg,
  author={Wang, Haiping and Liu, Yuan and Hu, Qingyong and Wang, Bing and Chen, Jianguo and Dong, Zhen and Guo, Yulan and Wang, Wenping and Yang, Bisheng},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={RoReg: Pairwise Point Cloud Registration with Oriented Descriptors and Local Rotations}, 
  year={2023},
  volume={},
  number={},
  pages={1-18},
  doi={10.1109/TPAMI.2023.3244951}}
}

🔗 Related Projects

Welcome to take a look at the homepage of our research group WHU-USI3DV ! We focus on 3D Computer Vision, particularly including 3D reconstruction, scene understanding, point cloud processing as well as their applications in intelligent transportation system, digital twin cities, urban sustainable development, and robotics.

YOHO is the early work of RoReg and we sincerely thank the contributing projects:

• EMVN for the group details;
• FCGF for the backbone;
• 3DMatch for the 3DMatch dataset;
• Predator for the 3DLoMatch dataset;
• ETH for the ETH dataset;
• PerfectMatch for organizing the 3DMatch and ETH dataset.