Implement frame skipping

[Alex-Christian Lazau]

Implementing frame skipping would increase performance by a lot, as usually not every frame a second is needed for specific tasks. I tried to do it like this:

skip_frames = 2
process_frame = cur_frames % skip_frames == 0
if process_frame:
    # Do the split_model part
else:
     # Just display the image

This approach leads to a lot of problems, where the split_model part uses and displays older images. As I am not familiar with Python threading, multiprocessing and queues, probably someone with more experience could implement this or give hints for the right direction.

[naisy]

Hi, TheAxelBoy

This method looks like a method for sequential processing of frames.

Since the WebcamVideoStream class in helper.py has been updated to the latest frame information in the thread, it can be said that this skip has already been done.

[Alex-Christian Lazau]

Hi,

So I just changed the code a little bit to apply processing on every x frame only.
I applied some changes in the session worker code:

# In __init__
self.process_frame = 

# In execution
while not self.sess_queue.empty():
    q = self.sess_queue.get(False)
    opts = q["opts"]
    feeds = q["feeds"]
    extras = q["extras"]
    if self.process_frame:
        if feeds None:
        ....
    self.result_queue.put(...)

# In put_sess_queue
def put_sess_queue(self, process_frame, opts, feeds=None, extras=None):
    self.process_frame = process_frame
    self.sess_queue.put(...)
    return

After that I added the new parameter in every call of cpu/gpu_worker.put_sess_queue(...):

skip_frames = 2  # Skip every second frame

# In the video stream loop
process_frame = (cur_frames % skip_frames == 0)
....
cpu/gpu_worker.put_sess_queue(process_frame, ...)

It is a quite naive approach and I used it in combination with Medianflow to track cars on a road, where not every frame needs to be run through the detector. I get about 50 fps (Jetson TX2) with skipping every second frame.

[naisy]

Hi @TheAxelBoy,

That sounds interesting!
I am not familiar with tracking technicks such as Medianflow, KFC and so on.

I looked at the OpenCV Tracking API. It seems easy to use.
I would like to try it as well.

[Gustav von Zitzewitz]

The OpenCV Tracking API is crap. i implemented it and it is slow as hell.
@TheAxelBoy i find your fram skipping approach interesting, would you like to post the whole code?

[Alex-Christian Lazau]

Hi @naisy

Although I don't like the OpenCV Tracking API, I built a multitracker using the medianflow single tracker as a base which works quite ok and has about 50 fps on its own. A better approach would probably be to write a Python wrapper for the Nvidia VisionWorks medianflow tracker.

@gustavz

I will fork the code and post the frame skipping code on my profile.