Discussion: can the method differentiate the background classes?

[Yu-Kai Chen]

Hi, thank you for providing this awsome work!

After reading the paper, I realized that we are highly relying on color, spatial, and the features extracted by the DINO algorithm.
So if we train a DINO model by the dataset that doesn't include background classes like road and sidewalk, and this two different classes shared the similar color and spatial features, can the eigen map still differentiate this two classes?
Thanks again for the discussion!!

Kevin

[Luke Melas-Kyriazi]

Thanks for the question and for starting this discussion.

Yes, I agree that the model relies heavily on local features like texture and color. This is a property that has been observed in a bunch of other settings as well (for example, Deep ViT Features as Dense Visual Descriptors from Tali Dekel's group).

As to whether it can differentiate the classes, it depends heavily on the individual setting. Sometimes it is surprisingly good at separating classes, and other times it groups different classes together.

Hope this helps!
Luke

[Yu-Kai Chen]

Thanks a lot for providing the good information and the discussion!

Kevin