InterMOT

Introduction

Multiple object tracking (MOT) is a significant task in achieving autonomous driving. Previous fusion methods usually fuse the top-level features after the backbones extract the features from different modalities. In this paper, we first introduce PointNet++ to obtain multi-scale deep representations of point cloud to make it adaptive to our proposed Interactive Feature Fusion between multi-scale features of images and point clouds. Specifically, through multi-scale interactive query and fusion between pixel-level and point-level features, our method, can obtain more distinguishing features to improve the performance of multiple object tracking.

For more details, please refer our paper Interactive Multi-scale Fusion of 2D and 3D Features for Multi-Object Tracking (TITS 2023) .

Install

This project is based on pytorch==1.1.0, because the following version does not support batch_size=1 for nn.GroupNorm.

We recommand you to build a new conda environment to run the projects as follows:

conda create -n intermot python=3.7 cython
conda activate intermot
conda install pytorch==1.1.0 torchvision==0.3.0 -c pytorch
conda install numba

Then install packages from pip:

pip install -r requirements.txt

Data

We provide the data split used in our paper in the data directory. You need to download and unzip the data from the KITTI Tracking Benchmark. You may follow Second for dataset preparation. Do remember to change the path in the configs.

The RRC detection results for training are obtained from MOTBeyondPixels. We use PermaTrack detection results provided by OC-SORT for the KITTI Tracking Benchmark. The detections are provided in data/detections already.

Usage

To train the model, you can run command

python main.py

Acknowledgement

This code benefits a lot from mmMOT and use the detection results provided by MOTBeyondPixels and OC-SORT . The GHM loss implementation is from GHM_Detection.

Citation

@article{wang2023interactive,
  title={Interactive Multi-Scale Fusion of 2D and 3D Features for Multi-Object Vehicle Tracking},
  author={Wang, Guangming and Peng, Chensheng and Gu, Yingying and Zhang, Jinpeng and Wang, Hesheng},
  journal={IEEE Transactions on Intelligent Transportation Systems},
  year={2023},
  publisher={IEEE}
}

IRMVLab / InterMOT