Handling total occlusions

[annahadji]

I'm trying to reproduce some of the results in the paper, and I'm interested in how the model deals with total occlusions.

For example, I notice in the extra qualitative results you provide, there is a moment where the person being tracked is fully occluded as someone else on a bike passes by (specifically here: https://youtu.be/R_Zae5N_hKw), and the occluded nodes no longer have the labels. I'm unsure how all of the labels disappeared? What happens to a node when its entirely occluded and goes out of sight?

In some initial results of running the model, it appears to predict that entirely occluded nodes (incorrectly) transition to neighbouring nodes or thereafter start tracking the occlusion, as opposed to not being predicted at all.

Thanks for any help in advance!

[A. Jabri]

Hi @annahadji, the algorithm used at inference time is a basic label propagation algorithm. More than one frame of context is provided for label propagation at each time step, meaning that the model can propagate labels around occlusion. Occluding objects that enter the scene (and do not have a match) take the background label, while objects that are newly dis-occluded take labels from context frames in which they are not occluded.