CMT for 3D Tracking in Point Clouds

Official implementation of the ECCV 2022 paper CMT: Context-Matching-Guided Transformer for 3D Tracking in Point Clouds

How to effectively match the target template features with the search area is the core problem in point-cloud-based 3D single object tracking. However, in the literature, most of the methods focus on devising sophisticated matching modules at point-level, while overlooking the rich spatial context information of points. To this end, we propose Context-Matching-Guided Transformer (CMT), a Siamese tracking paradigm for 3D single object tracking. In this work, we first leverage the local distribution of points to construct a horizontally rotation-invariant contextual descriptor for both the template and the search area. Then, a novel matching strategy based on shifted windows is designed for such descriptors to effectively measure the template-search contextual similarity. Furthermore, we introduce a target-specific transformer and a spatial-aware orientation encoder to exploit the target-aware information in the most contextually relevant template points, thereby enhancing the search feature for a better target proposal. We conduct extensive experiments to verify the merits of our proposed CMT and report a series of new state-of-the-art records on three widely-adopted datasets.

Setup

Installation

Create the environment

git clone https://github.com/jasongzy/CMT.git
cd CMT
conda create -n cmt python=3.8
conda activate cmt

Install pytorch

conda install pytorch torchvision cudatoolkit=11.3 -c pytorch

Install other dependencies:
```
pip install -r requirement.txt
```

Datasets

Please follow the setup guide of Open3DSOT.

Citation

@inproceedings{guo2022cmt,
  title={CMT: Context-Matching-Guided Transformer for 3D Tracking in Point Clouds},
  author={Guo, Zhiyang and Mao, Yunyao and Zhou, Wengang and Wang, Min and Li, Houqiang},
  booktitle={ECCV},
  year={2022}
}

Acknowledgment

This repo is built upon BAT.
Thank Erik Wijmans for his pytorch implementation of PointNet++.
Thank the implementation of PointSIFT in 3DNetworksPytorch.

jasongzy / CMT