Dual-Cross

This is the code related to "Cross-Domain and Cross-Modal Knowledge Distillation in Domain Adaptation for 3D Semantic Segmentation" (ACMMM 2022).

Paper

Cross-Domain and Cross-Modal Knowledge Distillation in Domain Adaptation for 3D Semantic Segmentation

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

If you find it helpful to your research, please cite as follows:

@inproceedings{li2022cross,
  title={Cross-Domain and Cross-Modal Knowledge Distillation in Domain Adaptation for 3D Semantic Segmentation},
  author={Li, Miaoyu and Zhang, Yachao and Xie, Yuan and Gao, Zuodong and Li, Cuihua and Zhang, Zhizhong and Qu, Yanyun},
  booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
  pages={3829--3837},
  year={2022}
}

Preparation (Refer to xMUDA)

Prerequisites

Tested with

PyTorch 1.4
CUDA 10.0
Python 3.8
SparseConvNet
nuscenes-devkit

Installation

As 3D network we use SparseConvNet. It requires to use CUDA 10.0 (it did not work with 10.1 when we tried). We advise to create a new conda environment for installation. PyTorch and CUDA can be installed, and SparseConvNet installed/compiled as follows:

$ conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
$ pip install --upgrade git+https://github.com/facebookresearch/SparseConvNet.git

$ cd xmuda
$ pip install -ve .

The -e option means that you can edit the code on the fly.

Datasets

NuScenes

Please download the Full dataset (v1.0) from the NuScenes website and extract it.

You need to perform preprocessing to generate the data for Dual-Cross.

Please edit the script xmuda/data/nuscenes/preprocess.py as follows and then run it.

root_dir should point to the root directory of the NuScenes dataset
out_dir should point to the desired output directory to store the pickle files

Usage

Training

You can run the training with:

$ python xmuda/train_dual.py --cfg=configs/nuscenes/day_night/xmuda.yaml

You can change the path OUTPUT_DIR in the config file xmuda.yaml.

Testing

You can provide which checkpoints you want to use for testing. We used the ones that performed best on the validation set during training (the best valiteration for 2D and 3D is shown at the end of each training). Note that @ will be replaced by the output directory for that config file. For example:

$ python xmuda/test.py --cfg=configs/nuscenes/day_night/xmuda.yaml  @/model_2d_065000.pth @/model_3d_095000.pth

You can also provide an absolute path without @.

Acknowledgements

Note that this code borrows from the xMUDA repo.

lilujunai / Dual-Cross