ICCV2021_Event_Frame_Tracking

Object Tracking by Jointly Exploiting Frame and Event Domain (ICCV 2021)

Jiqing Zhang, Xin Yang, Yingkai Fu, Xiaopeng Wei, Baocai Yin, Bo Dong

The code is built on visionml/pytracking and tested on Ubuntu 18.04 environment with RTX 3090 GPUs.

Abstract

Inspired by the complementarity between conventional frame-based and bio-inspired event-based cameras, we propose a multi-modal based approach to fuse visual cues from the frame- and event-domain to enhance the single object tracking performance, especially in degraded conditions (e.g., scenes with high dynamic range, low light, and fast motion objects). The proposed approach can effectively and adaptively combine meaningful information from both domains. Our approach’s effectiveness is enforced by a novel designed cross-domain attention schemes, which can effectively enhance features based on self- and cross-domain attention schemes; The adaptiveness is guarded by a specially designed weighting scheme, which can adaptively balance the contribution of the two domains. To exploit event-based visual cues in single-object tracking, we construct a largescale frame-event-based dataset, which we subsequently employ to train a novel frame-event fusion based model. Extensive experiments show that the proposed approach outperforms state-of-the-art frame-based tracking methods by at least 10.4% and 11.9% in terms of representative success rate and precision rate, respectively. Besides, the effectiveness of each key component of our approach is evidenced by our thorough ablation study.

Requirements

CUDA == 11.1
Python == 3.8
Pytorch == '1.7.0+cu110'

Test

Download test dataset
Download the pretrained model, and put it into pytracking_fe108/pytracking/networks
Change your own path in pytracking_fe108/pytracking/evaluation/local.py
run python run_tracker.py dimp prdimp18 --dataset eotb --sequence val, the predicted bbox will be saved in pytracking_fe108/pytracking/tracking_results. Using jupyter in notebooks to see the SR and PR scores.

Citation

If you use this code, please cite:

@inproceedings{zhang2021object,
  title={Object Tracking by Jointly Exploiting Frame and Event Domain},
  author={Zhang, Jiqing and Yang, Xin and Fu, Yingkai and Wei, Xiaopeng and Yin, Baocai and Dong, Bo},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={13043--13052},
  year={2021}
}

Acknowledgments

Thanks for the great visionml/pytracking module.

Jee-King / ICCV2021_Event_Frame_Tracking