CASR-HKU / REMOT-FPGA-22

REMOT

REMOT: A Hardware-Software Architecture for Attention-Guided Multi-Object Tracking with Dynamic Vision Sensors on FPGAs

Citation

@article{yizhao2021remot,
    title = "{REMOT: A Hardware-Software Architecture for Attention-Guided Multi-Object Tracking with Dynamic Vision Sensors on FPGAs}",
    author = {Yizhao, Gao and Song, Wang and So, Hayden K.-H.},
    journal = {The 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays},
    year = {2022}
}

Introduction

REMOT is a hardware/software architecture for Multi-Object Tracking using Dynamic Vision Sensors on FPGA. It's designed around the concept of an attention unit (AU). Each AU will only pay attention to a specific region of interest, which is designed to changed as the object moves. In REMOT, a layer of parallel AUs is implemented on FPGA to collectively process the stream of asynchronous events from a DVS.

Outlines

• software: Python implementation of AU for Multi-Object Tracking
• hardware: HLS code and vivado scripts for 3 different implementations: FUll-AMAP, HASH-AMAP, and FIFO-ONLY. Python drive for PYNQ overlay is also provided for PYNQ-Z2 and Ultra96 development boards.

Dependency

The repo has been verified on:

• Ubuntu 18.04
• Vivado 2020.2
• Matlab R2021b
• python 3.7
• PYNQ v2.6 image

Vivado 2020.2

• We provide scripts with Vivado 2020.2 to build bitstream. If using a different version, some mannul intervention might be needed.

    source <vivado_root/2020.2/settings64.sh>

Python Environment Installation

• Use conda to install python libraries:

    conda create -n au python=3.7
    conda activate au
    pip install -r requirements.txt

• Install Matlab Engine API for python

    cd <matlab_root/extern/engines/python>
    python setup.py install