An optical flow and Kalman Filter based multi-ojbect tracker
Object detection is slow, especially for embedded platforms. Tracking algorithm implementations in OpenCV3 contrib does not work well
for multi-object tracking, the processing time increases linearly with the number of trackers. And they are all long-term tracking
oriented. When we have detection every once a while, we do not need the trackers to be that accurate, and we need high speed tracking.
So I implemented this optical-flow and kalman filter based multi-object tracker, ~50ms processing time for a 640x480 frame tested on Odroid XU4.
lk_tracker.cppcontains the tracker implementation, including
LkTrackeras an individual tracker and
TrackerManagerwhich manages all trackers
object_detection.cppcontains an OpenCV face detector, you can modify the class to use other object detection.
lk_tracker_test.cppcontains a simple test in a single-threaded setting. It requires opencv3 contrib to draw the initial bounding box. Modify that if you do not have OpenCV 3 contrib. Or simply commment it out in CMakeLists.txt if you do not want it.
det_and_track.cppcontatins the setting to use our tracker and a face detector in a multi-threaded setting. The face detector here can be replaced with any detector you like.
main.cppcontains the usage of the detection and tracking in multi-thread setting.
- Modify the OpenCV path in CMakeLists.txt.
- Modify the detection_sleep_time_ and track_sleep_time_ to 0 when using with embedded platforms.
- Use Kalman Filter or not:
USE_KF_in TrackerManger's constructor in
lk_tracker.cpp. If true, the Kalman Filter will be applied, and the trackers will be more robust against oclussion, however it may also cause drifting problem.
- Matching cost threshold:
lk_tracker.cpp, everytime we get detection results from the detector, we will match all the bouding boxes with the existing trackers using Hungarian algorithm. However, sometimes even when a detection result is matched with an existing trackers, it may still not be ideal. Thus we use the
COST_THRESHOLD_to get rid of this match.
- MIN_ACCEPT_FRAMES_ and MIN_REJECT_FRAMES_ in lk_tracker.cpp
If in continuous MIN_ACCPET_FRMAES_ frames, a tracker can be matched with a detection result, then this tracker is 'qualified' as a tracker to a most-likely real face. If in continuous MIN_REJECT_FRAMES_ frames, a tracker cannot be matched with detection result, then this tracker is most-likely lost tracking and will be removed.
- Detection and tracking frequency:
track_sleep_time_together with the performanmce of you computer determines the detection and tracking frequency. Let's say it takes 100ms for detection for each frame, and you set detection_sleep_time_ to 150ms, then you will get detection results every 250ms. The smaller the value means the better results you can get but more load on your CPU. Usually if you want to get the best performance in embedded environment (Raspberry Pi or Odroid), just set them to 0.
Checkout https://github.com/zlingkang/mtcnn_face_detection_and_tracking/tree/master to see how to replace the OpenCV face detector used in this repo to a deep learning based face detector which is much more accurate.
- OpenCV LK optical flow
- Munkres (Hungarian) algorithm implementation: https://github.com/soimy/munkres-opencv