OpenCV MOSSE tracker crashes after frames with no objects detected

Question

OpenCV MOSSE tracker crashes after frames with no objects detected

84679452 opened this issue 2 years ago · comments

Operating System: Windows 11 21H2 22000.708
Python 3.8.13
PeekingDuck 1.2.1

Steps to reproduce:

Create new PeekingDuck environment

conda create -n pkd python=3.8
conda activate pkd
pip install -U peekingduck

Optionally install CUDA libraries.

conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0

Initialize peekingduck folder and edit pipeline_config.yml following template from tutorial.

pipeline_config.yml

nodes:
- input.visual:
    source: videoplayback.mp4
- model.efficientdet:
    detect_ids: ["car"]
- dabble.tracking:
    tracking_type: "mosse"
- dabble.statistics:
    maximum: obj_attrs["ids"]
- draw.bbox
- draw.tag:
    show: ["ids"]
- draw.legend:
    show: ["cum_max"]
- output.screen

Using test video downloaded from YouTube, PeekingDuck crashes at the point in the video right before entering the garage, with the error log below:

(pkd_test) C:\Users\test\Desktop\test>peekingduck run
2022-06-13 16:14:15 peekingduck.declarative_loader  INFO:  Successfully loaded pipeline file.
2022-06-13 16:14:15 peekingduck.declarative_loader  INFO:  Initializing input.visual node...
2022-06-13 16:14:15 peekingduck.declarative_loader  INFO:  Config for node input.visual is updated to: 'source': videoplayback.mp4
2022-06-13 16:14:15 peekingduck.pipeline.nodes.input.visual  INFO:  Input size: 640 by 360
2022-06-13 16:14:15 peekingduck.declarative_loader  INFO:  Initializing model.efficientdet node...
2022-06-13 16:14:17 peekingduck.declarative_loader  INFO:  Config for node model.efficientdet is updated to: 'detect_ids': [2]
2022-06-13 16:14:22 peekingduck.pipeline.nodes.model.efficientdet_d04.efficientdet_files.detector  INFO:  EfficientDet model loaded with following configs:
        Model type: D0
        IDs being detected: [2]
        Score threshold: 0.3
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Initializing dabble.tracking node...
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Config for node dabble.tracking is updated to: 'tracking_type': mosse
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Initializing dabble.statistics node...
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Config for node dabble.statistics is updated to: 'maximum': obj_attrs["ids"]
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Initializing draw.bbox node...
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Initializing draw.tag node...
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Config for node draw.tag is updated to: 'show': ['ids']
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Initializing draw.legend node...
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Config for node draw.legend is updated to: 'show': ['cum_max']
2022-06-13 16:14:22 peekingduck.declarative_loader  INFO:  Initializing output.screen node...
2022-06-13 16:14:41 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 100
2022-06-13 16:14:54 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 200
2022-06-13 16:15:04 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 300
2022-06-13 16:15:12 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 400
2022-06-13 16:15:16 root  ERROR:  Traceback (most recent call last):
File "C:\Users\test\anaconda3\envs\pkd_test\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
File "C:\Users\test\anaconda3\envs\pkd_test\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
File "C:\Users\test\anaconda3\envs\pkd_test\Scripts\peekingduck.exe\__main__.py", line 7, in <module>
    sys.exit(cli())
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\cli.py", line 327, in run
    runner.run()
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\runner.py", line 127, in run
    outputs = node.run(inputs)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\tracking.py", line 81, in run
    track_ids = self.tracker.track_detections(inputs)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\detection_tracker.py", line 72, in track_detections
    track_ids = self.tracker.track_detections(inputs)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\tracking_files\opencv_tracker.py", line 62, in track_detections
    obj_track_ids = self._match_and_track(frame, tlwhs)
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\tracking_files\opencv_tracker.py", line 103, in _match_and_track
    ious = iou_candidates(bbox, np.array(prev_tracked_bboxes))
File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\tracking_files\utils.py", line 36, in iou_candidates
    candidates_tl = candidates[:, :2]

2022-06-13 16:15:16 root  ERROR:  IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

Additional Comments:
Bug should be reproducible with any video where there are frames with no objects detected followed by frames where objects appear again. Alternatively, an easier way would be to use a web camera and change the pipeline to detect "persons".

Quick Workaround:
Initialize prev_tracked_bboxes to empty 2D array if empty array [ ] and add additional sanity check to def _match_and_track.

peekingduck\pipeline\nodes\dabble\trackingv1\tracking_files\opencv_tracker.py

def _match_and_track(self, frame: np.ndarray, bboxes: np.ndarray) -> List[int]:
    """Matches detections to tracked bboxes, creates a new track if no
    match is found.

    Args:
        frame (np.ndarray): Input video frame.
        bboxes (np.ndarray): Detection bounding boxes with (t, l, w, h)
            format where (t, l) is the coordinate of the top-left corner,
            w is the width, and h is the height of the bounding box
            respectively.

    Returns:
        (List[int]): A list of track IDs for the detections in the current
            frame.
    """
    prev_tracked_bboxes = [track.bbox for _, track in self.tracks.items()]
    matching_dict = {}
    if not len(prev_tracked_bboxes):
        prev_tracked_bboxes = np.empty([0,4])
    
    for bbox in bboxes:
        ious = iou_candidates(bbox, np.array(prev_tracked_bboxes))
        matching_dict[tuple(bbox)] = (
            ious.argmax() if (len(ious) and max(ious) >= self.iou_threshold) else None
        )
    
    track_ids = []
    for bbox, matched_id in matching_dict.items():
        if matched_id is None:
            self._initialize_tracker(frame, np.array(bbox))
            track_ids.append(list(self.tracks)[-1])
        else:
            track_ids.append(list(self.tracks)[matched_id])
    
    return track_ids

liyier90 · Answer 1 · Tue Jun 14 2022 09:30:53 GMT+0800 (China Standard Time)

Thanks for the report! A fix for this has been submitted and will be in our next release after it's merged.

84679452 · Answer 2 · Tue Jun 14 2022 09:53:47 GMT+0800 (China Standard Time)

Hi, with the fix #662 the processing is able to pass the point of entering the garage but still crashes upon exiting the garage.
The log is as below:

>>> peekingduck run
2022-06-14 09:48:25 peekingduck.declarative_loader  INFO:  Successfully loaded pipeline file.
2022-06-14 09:48:25 peekingduck.declarative_loader  INFO:  Initializing input.visual node...
2022-06-14 09:48:25 peekingduck.declarative_loader  INFO:  Config for node input.visual is updated to: 'source': videoplayback.mp4
2022-06-14 09:48:25 peekingduck.pipeline.nodes.input.visual  INFO:  Input size: 640 by 360
2022-06-14 09:48:25 peekingduck.declarative_loader  INFO:  Initializing model.yolo node...
2022-06-14 09:48:27 peekingduck.declarative_loader  INFO:  Config for node model.yolo is updated to: 'detect_ids': [2]
2022-06-14 09:48:27 peekingduck.pipeline.nodes.model.yolov4.yolo_files.detector  INFO:  YOLO model loaded with following configs:
        Model type: v4tiny,
        Input resolution: (416, 416),
        IDs being detected: [2]
        Max detections per class: 50,
        Max total detections: 50,
        IOU threshold: 0.5,
        Score threshold: 0.2
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Initializing dabble.tracking node...
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Config for node dabble.tracking is updated to: 'tracking_type': mosse
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Initializing dabble.statistics node...
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Config for node dabble.statistics is updated to: 'maximum': obj_attrs["ids"]
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Initializing draw.bbox node...
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Initializing draw.tag node...
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Config for node draw.tag is updated to: 'show': ['ids']
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Initializing draw.legend node...
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Config for node draw.legend is updated to: 'show': ['cum_max']
2022-06-14 09:48:29 peekingduck.declarative_loader  INFO:  Initializing output.screen node...
2022-06-14 09:48:33 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 100
2022-06-14 09:48:34 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 200
2022-06-14 09:48:36 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 300
2022-06-14 09:48:38 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 400
2022-06-14 09:48:39 peekingduck.pipeline.nodes.input.visual  INFO:  Frames Processed: 500
2022-06-14 09:48:39 root  ERROR:  Traceback (most recent call last):
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
   File "C:\Users\test\anaconda3\envs\pkd_test\Scripts\peekingduck.exe\__main__.py", line 7, in <module>
    sys.exit(cli())
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\cli.py", line 327, in run
    runner.run()
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\runner.py", line 127, in run
    outputs = node.run(inputs)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\tracking.py", line 81, in run
    track_ids = self.tracker.track_detections(inputs)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\detection_tracker.py", line 72, in track_detections
    track_ids = self.tracker.track_detections(inputs)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\tracking_files\opencv_tracker.py", line 62, in track_detections
    obj_track_ids = self._match_and_track(frame, tlwhs)
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\tracking_files\opencv_tracker.py", line 103, in _match_and_track
    ious = iou_candidates(bbox, np.array(prev_tracked_bboxes))
   File "C:\Users\test\anaconda3\envs\pkd_test\lib\site-packages\peekingduck\pipeline\nodes\dabble\trackingv1\tracking_files\utils.py", line 36, in iou_candidates
    candidates_tl = candidates[:, :2]

2022-06-14 09:48:39 root  ERROR:  IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

liyier90 · Answer 3 · Tue Jun 14 2022 11:35:47 GMT+0800 (China Standard Time)

Made another update to #662 and was to process the video to completion on my machine. Could you please try and see if it fixes the issue for you?

84679452 · Answer 4 · Tue Jun 14 2022 11:43:01 GMT+0800 (China Standard Time)

Hi, tried the new fix and was able to pass the garage scene. Didn't test the entire video, but issue should be fixed now. Thanks!