Visual Prompt Multi-Modal Tracking [CVPR2023]

Official implementation of ViPT, including models and training&testing codes.

Models & Raw Results (Google Driver) Models & Raw Results (Baidu Driver: vipt)

🔥🔥🔥 This work proposes ViPT, a new prompt-tuning framework for multi-modal tracking.

Tracking in RGB + Depth scenarios:
Tracking in RGB + Thermal scenarios:
Tracking in RGB + Event scenarios:

News

[Mar 20, 2023]

We release codes, models and raw results.
Thanks for your star 😝😝😝.

[Feb 28, 2023]

ViPT is accepted to CVPR2023.

Introduction

🔥 A new unified visual prompt multi-modal tracking framework (e.g. RGB-D, RGB-T, and RGB-E Tracking).
ViPT has high performance on multiple multi-modal tracking tasks.
ViPT is with high parameter-efficient tuning, containing only 0.84M trainable parameters (<1%).
We expect ViPT can attract more attention to prompt learning 🔥 for further research of multi-modal tracking.

Results

On RGB-D tracking benchmarks

On RGB-T tracking benchmarks

On RGB-E tracking benchmark

Usage

Installation

Create and activate a conda environment:

conda create -n vipt python=3.7
conda activate vipt

Install the required packages:

bash install_vipt.sh

Data Preparation

Put the training datasets in ./data/. It should look like:

$<PATH_of_ViPT>
-- data
    -- DepthTrackTraining
        |-- adapter02_indoor
        |-- bag03_indoor
        |-- bag04_indoor
        ...
    -- LasHeR/train/trainingset
        |-- 1boygo
        |-- 1handsth
        ...
    -- VisEvent/train
        |-- 00142_tank_outdoor2
        |-- 00143_tank_outdoor2
        ...
        |-- trainlist.txt

Path Setting

Run the following command to set paths:

cd <PATH_of_ViPT>
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output

You can also modify paths by these two files:

./lib/train/admin/local.py  # paths for training
./lib/test/evaluation/local.py  # paths for testing

Training

Dowmload the pretrained foundation model (OSTrack) and put it under ./pretrained/.

bash train_vipt.sh

You can train models with various modalities and variants by modifying train_vipt.sh.

Testing

For RGB-D benchmarks

[DepthTrack Test set & VOT22_RGBD]
These two benchmarks are evaluated using VOT-toolkit.
You need to put the DepthTrack test set to./Depthtrack_workspace/ and name it 'sequences'.
You need to download the corresponding test sequences at./vot22_RGBD_workspace/.

bash eval_rgbd.sh

For RGB-T benchmarks

[LasHeR & RGBT234]
Modify the <DATASET_PATH> and <SAVE_PATH> in./RGBT_workspace/test_rgbt_mgpus.py, then run:

bash eval_rgbt.sh

We refer you to LasHeR Toolkit for LasHeR evaluation, and refer you to MPR_MSR_Evaluation for RGBT234 evaluation.

For RGB-E benchmark

[VisEvent]
Modify the <DATASET_PATH> and <SAVE_PATH> in./RGBE_workspace/test_rgbe_mgpus.py, then run:

bash eval_rgbe.sh

We refer you to VisEvent_SOT_Benchmark for evaluation.

Bixtex

If you find ViPT is helpful for your research, please consider citing:

@inproceedings{ViPT,
  title={Visual Prompt Multi-Modal Tracking},
  author={Jiawen, Zhu and Simiao, lai and Xin, Chen and Wang, Dong and Lu, Huchuan},
  booktitle={CVPR},
  year={2023}
}

Acknowledgment

This repo is based on OSTrack which is an excellent work.
We thank for the PyTracking library, which helps us to quickly implement our ideas.

Contact

If you have any question, feel free to email jiawen@mail.dlut.edu.cn. ^_^

jiawen-zhu / ViPT