EINet

The official code and benchmark for our paper: Explicit Interaction for Fusion-Based Place Recognition.

This work has been accepted by IROS 2024 🎉

Jingyi Xu, Junyi Ma, Qi Wu, Zijie Zhou, Yue Wang, Xieyuanli Chen, Wenxian Yu, Ling Pei*.

Installation

We follow the installation instructions of our codebase LCPR, which are also posted here.

Create a conda virtual environment and activate it

git clone git@github.com:BIT-XJY/EINet.git
cd EINet
conda create -n EINet python=3.8
conda activate EINet

Install other dependencies

pip install -r requirements.txt

Data Download

Please download the offical nuScenes dataset, and link it to the data folder.
nuscenes_occ_infos_train.pkl, and nuscenes_occ_infos_val.pkl are also provided by the previous work. We also use it to train the EINet.

Note that the download data structure should be like:

nuscenes
├─ raw_data
│    ├─ maps
│    │    ├─ ...
│    ├─ samples
│    │    ├─ CAM_BACK
│    │    ├─ CAM_BACK_LEFT
│    │    ├─ CAM_BACK_RIGHT
│    │    ├─ CAM_FRONT
│    │    ├─ CAM_FRONT_LEFT
│    │    ├─ CAM_FRONT_RIGHT
│    │    ├─ LIDAR_TOP
│    │    ├─ RADAR_BACK_LEFT
│    │    ├─ RADAR_BACK_RIGHT
│    │    ├─ RADAR_FRONT
│    │    ├─ RADAR_FRONT_LEFT
│    │    ├─ RADAR_FRONT_RIGHT
│    ├─ sweeps
│    │    ├─ CAM_BACK
│    │    ├─ CAM_BACK_LEFT
│    │    ├─ CAM_BACK_RIGHT
│    │    ├─ CAM_FRONT
│    │    ├─ CAM_FRONT_LEFT
│    │    ├─ CAM_FRONT_RIGHT
│    │    ├─ LIDAR_TOP
│    │    ├─ RADAR_BACK_LEFT
│    │    ├─ RADAR_BACK_RIGHT
│    │    ├─ RADAR_FRONT
│    │    ├─ RADAR_FRONT_LEFT
│    │    ├─ RADAR_FRONT_RIGHT
│    ├─ v1.0-test
│    │    ├─ attribute.json
│    │    ├─ calibrated_sensor.json
│    │    ├─ ...
│    ├─ v1.0-traninval
│    │    ├─ attribute.json
│    │    ├─ calibrated_sensor.json
│    │    ├─ ...

NUSC-PR

We propose the NUSC-PR benchmark to split nuScenes datasets with self-supervised and supervised learning schemes.

Self-supervised Data Preparation

Extract basic information from nuScenes datasets, and split query and database for NUSC-PR.

cd NUSC-PR
cd self_supervised
python generate_basic_infos.py
python split_dataset.py
cd ..

The data structure with a self-supervised learning scheme should be like:

self_supervised_data
├─ generate_basic_infos
│    ├─ nuscenes_infos-bs.pkl
│    ├─ nuscenes_infos-shv.pkl
│    ├─ nuscenes_infos-son.pkl
│    ├─ nuscenes_infos-sq.pkl
│    ├─ nuscenes_infos.pkl
├─ split_dataset
│    ├─ all_train_query_pos_neg_index_in_infos.pkl
│    ├─ bs_db_index_in_infos.npy
│    ├─ bs_test_query_gt_index_in_infos.pkl
│    ├─ bs_train_query_pos_neg_index_in_infos.pkl
│    ├─ shv_db_index_in_infos.npy
│    ├─ shv_test_query_gt_index_in_infos.pkl
│    ├─ shv_train_query_pos_neg_index_in_infos.pkl
│    ├─ son_db_index_in_infos.npy
│    ├─ son_test_query_gt_index_in_infos.pkl
│    ├─ son_train_query_pos_neg_index_in_infos.pkl
│    ├─ sq_db_index_in_infos.npy
│    ├─ sq_test_query_gt_index_in_infos.pkl
│    ├─ sq_train_query_pos_neg_index_in_infos.pkl

Supervised Data Preparation

Extract basic information from nuScenes datasets, and split query and database for NUSC-PR.

cd supervised
python generate_basic_infos.py
python split_dataset.py
python select_pos_neg_samples_by_dis.py
python generate_selected_indicies.py
cd ..
cd ..

The data structure with a supervised learning scheme should be like:

supervised_data
├─ generate_basic_infos
│    ├─ nuscenes_infos-bs.pkl
│    ├─ nuscenes_infos-shv.pkl
│    ├─ nuscenes_infos-son.pkl
│    ├─ nuscenes_infos-sq.pkl
│    ├─ nuscenes_infos.pkl
├─ generate_selected_indicies
│    ├─ bs_db_index_in_infos.npy
│    ├─ bs_test_query_gt_index_in_infos.pkl
│    ├─ bs_train_query_pos_neg_index_in_infos.pkl
│    ├─ shv_db_index_in_infos.npy
│    ├─ shv_test_query_gt_index_in_infos.pkl
│    ├─ shv_train_query_pos_neg_index_in_infos.pkl
│    ├─ son_db_index_in_infos.npy
│    ├─ son_test_query_gt_index_in_infos.pkl
│    ├─ son_train_query_pos_neg_index_in_infos.pkl
│    ├─ sq_db_index_in_infos.npy
│    ├─ sq_test_query_gt_index_in_infos.pkl
│    ├─ sq_train_query_pos_neg_index_in_infos.pkl
├─ select_pos_neg_samples_by_dis
│    ├─ bs_test_query_gt_tokens.pkl
│    ├─ bs_train_query_pos_neg_tokens.pkl
│    ├─ shv_test_query_gt_tokens.pkl
│    ├─ shv_train_query_pos_neg_tokens.pkl
│    ├─ son_test_query_gt_tokens.pkl
│    ├─ son_train_query_pos_neg_tokens.pkl
│    ├─ sq_test_query_gt_tokens.pkl
│    ├─ sq_train_query_pos_neg_tokens.pkl
├─ split_dataset
│    ├─ bs_db_sample_token.npy
│    ├─ bs_db.npy
│    ├─ bs_sample_token.npy
│    ├─ bs_test_query_sample_token.npy
│    ├─ bs_test_query.npy
│    ├─ bs_train_query_sample_token.npy
│    ├─ bs_train_query.npy
│    ├─ bs_val_query_sample_token.npy
│    ├─ bs_val_query.npy
│    ├─ shv_db_sample_token.npy
│    ├─ shv_db.npy
│    ├─ shv_sample_token.npy
│    ├─ shv_test_query_sample_token.npy
│    ├─ shv_test_query.npy
│    ├─ shv_train_query_sample_token.npy
│    ├─ shv_train_query.npy
│    ├─ shv_val_query_sample_token.npy
│    ├─ shv_val_query.npy
│    ├─ son_db_sample_token.npy
│    ├─ son_db.npy
│    ├─ son_sample_token.npy
│    ├─ son_test_query_sample_token.npy
│    ├─ son_test_query.npy
│    ├─ son_train_query_sample_token.npy
│    ├─ son_train_query.npy
│    ├─ son_val_query_sample_token.npy
│    ├─ son_val_query.npy
│    ├─ sq_db_sample_token.npy
│    ├─ sq_db.npy
│    ├─ sq_sample_token.npy
│    ├─ sq_test_query_sample_token.npy
│    ├─ sq_test_query.npy
│    ├─ sq_train_query_sample_token.npy
│    ├─ sq_train_query.npy
│    ├─ sq_val_query_sample_token.npy
│    ├─ sq_val_query.npy

TODO

Release the paper
Release the benchmark NUSC-PR code for EINet
Release the source code for EINet
Release our pretrained baseline model

Acknowledgement

We thank the fantastic works LCPR, ManyDepth, and AutoPlace for their pioneer code release, which provide codebase for this work.

BIT-XJY / EINet