CMRNet-Fixed

This repository is a modified version of CMRNet

Modification

Fix UnicodeEncodeError
Change all uv,uv2 to be contiguous
Change all tensors to be on the same device

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

News

Check out our new paper "CMRNet++: Map and Camera Agnostic Monocular Visual Localization in LiDAR Maps":

2022/05/19

Updated for later version of PyTorch 1.4+, and CUDA 11.x

2020/06/24

We released the pretrained weights, see Pretrained Model.

2020/05/11

We released the SLAM ground truth files, see Local Maps Generation.

Multi-GPU training.

Added requirements.txt

Code

PyTorch implementation of CMRNet.

This code is a provided "as is", without warranty of any kind. This version only works on GPUs (no CPU version available).

Tested on:

Ubuntu 16.04/18.04
python 3.6
cuda 9/10/11.x
pytorch 1.0.1/1.10

Dependencies (this list is not complete):

sacred
mathutils (use this version: https://gitlab.com/m1lhaus/blender-mathutils)
openCV (for visualization)
open3d 0.7 (only for maps preprocessing)
pykitti (only for maps preprocessing)

Installation

Install CUDA, PyTorch, CuPy. Make sure to use the correct cuda version for all the packages.

⚠️ For CUDA 11.x please uncomment line 17 in models/CMRNet/correlation_package/setup.py setup.py#L17

Install the prerequisite packages:

pip install -r requirements.txt

And finally, install the correlation_cuda and the visibility package:

cd models/CMRNet/correlation_package/
python setup.py install
cd ../../..
python setyp.py install

It is recommended to use a dedicated conda environment

Data

We trained and tested CMRNet on the KITTI odometry sequences 00, 03, 05, 06, 07, 08, and 09.

We used a LiDAR-based SLAM system to generate the ground truths.

The Data Loader requires a local point cloud for each camera frame, the point cloud must be expressed with respect to the camera_2 reference frame, BUT (very important) with a different axes representation: X-forward, Y-right, Z-down.

For reading speed and file size we decided to save the point clouds as h5 files.

The directory structure should looks like:

KITTI_ODOMETRY
└── sequences
    ├── 00
    │   ├── image_2
    │   │   ├── 000000.png
    │   │   ├── 000001.png
    │   │   ├── ...
    │   │   └── 004540.png
    │ 	├── velodyne
    │   │   ├── 000000.bin
    │   │   ├── 000001.bin
    │   │   ├── ...
    │   │   └── 004540.bin
    │   ├── local_maps
    │   │   ├── 000000.h5
    │   │   ├── 000001.h5
    │   │   ├── ...
    │   │   └── 004540.h5
    │    ├── poses.csv
    │		└── calib.txt
    ├── 03		
    │   ├── image_2
    │   │   ├── 000000.png
    │   │   ├── 000001.png
    │   │   ├── ...
    │   │   └── 000800.png
    │ 	├── velodyne
    │   │   ├── 000000.bin
    │   │   ├── 000001.bin
    │   │   ├── ...
    │   │   └── 000800.bin
    │   ├── local_maps
    │   │   ├── 000000.h5
    │   │   ├── 000001.h5
    │   │   ├── ...
    │   │   └── 000800.h5
    │   ├── poses.csv
    │		└── calib.txt
    └── ...

Local Maps Generation

To generate the h5 files, use the script preprocess/kitti_maps.py with the ground truth files in data/.

In the sequence 08, the SLAM failed to detect a loop closure, so the poses are not coherent around that closure. Therefore, we splitted the map at frame 3000, so to have two coherent maps for that sequence.

python preprocess/kitti_maps.py --sequence 00 --kitti_folder ./KITTI_ODOMETRY/
python preprocess/kitti_maps.py --sequence 03 --kitti_folder ./KITTI_ODOMETRY/
python preprocess/kitti_maps.py --sequence 05 --kitti_folder ./KITTI_ODOMETRY/
python preprocess/kitti_maps.py --sequence 06 --kitti_folder ./KITTI_ODOMETRY/
python preprocess/kitti_maps.py --sequence 07 --kitti_folder ./KITTI_ODOMETRY/
python preprocess/kitti_maps.py --sequence 08 --kitti_folder ./KITTI_ODOMETRY/ --end 3000
python preprocess/kitti_maps.py --sequence 08 --kitti_folder ./KITTI_ODOMETRY/ --start 3000
python preprocess/kitti_maps.py --sequence 09 --kitti_folder ./KITTI_ODOMETRY/

Single Iteration example

Training:

python main_visibility_CALIB.py with batch_size=24 data_folder=./KITTI_ODOMETRY/ epochs=300 max_r=10 max_t=2 BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0

Evaluation:

python evaluate_iterative_single_CALIB.py with test_sequence=00 maps_folder=local_maps data_folder=./KITTI_ODOMETRY/ weight="['./checkpoints/weights.tar']"

Iterative refinement example

Training

python main_visibility_CALIB.py with batch_size=24 data_folder=./KITTI_ODOMETRY/sequences/ epochs=300 max_r=10 max_t=2   BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0
python main_visibility_CALIB.py with batch_size=24 data_folder=./KITTI_ODOMETRY/sequences/ epochs=300 max_r=2  max_t=1   BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0
python main_visibility_CALIB.py with batch_size=24 data_folder=./KITTI_ODOMETRY/sequences/ epochs=300 max_r=2  max_t=0.6 BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0

Evaluation

python evaluate_iterative_single_CALIB.py with test_sequence=00 maps_folder=local_maps data_folder=./KITTI_ODOMETRY/sequences/ weight="['./checkpoints/iter1.tar','./checkpoints/iter2.tar','./checkpoints/iter3.tar']"

Pretrained Model

The weights for the three iterations, trained on the sequences 03, 05, 06, 07, 08 and 09 are available here: Iteration 1 Iteration 2 Iteration 3

Results:

	Median Transl. error	Median Rotation. error
Iteration 1	0.46 m	1.60°
Iteration 2	0.25 m	1.14°
Iteration 3	0.20 m	0.97°

Paper

"CMRNet: Camera to LiDAR-Map Registration"

If you use CMRNet, please cite:

@InProceedings{cattaneo2019cmrnet,
  author={Cattaneo, Daniele and Vaghi, Matteo and Ballardini, Augusto Luis and Fontana, Simone and Sorrenti, Domenico Giorgio and Burgard, Wolfram},
  booktitle={2019 IEEE Intelligent Transportation Systems Conference (ITSC)},
  title={CMRNet: Camera to LiDAR-Map Registration},
  year={2019},
  pages={1283-1289},
  doi={10.1109/ITSC.2019.8917470},
  month={Oct}
}

If you use the ground truths, please also cite:

@INPROCEEDINGS{Caselitz_2016, 
  author={T. {Caselitz} and B. {Steder} and M. {Ruhnke} and W. {Burgard}}, 
  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
  title={Monocular camera localization in 3D LiDAR maps}, 
  year={2016},
  pages={1926-1931},
  doi={10.1109/IROS.2016.7759304}
}

Acknowledgments

correlation_package was taken from flownet2

PWCNet.py is a modified version of the original PWC-DC network

Contacts

Daniele Cattaneo (cattaneo@informatik.uni-freiburg.de or d.cattaneo10@campus.unimib.it)

zang09 / CMRNet_Fixed