Using selfsim_fc layer for label propagation

[Vadim Kantorov]

@ajabri By chance, have you tried using the layer from selfsim_fc head for label propagation? In appendix G you mention that res4-features perform worse than res3. But what about selfsim_fc? It is located even closer to the loss function, does it perform even worse than res4?

Thanks!

[A. Jabri]

I haven't tried this, but note one detail; selfsim_fc parameters are learned at training time on top of pooled patch embeddings (since we train with patches). So it makes less sense to apply that layer on elements of the dense feature map at test time.

[Vadim Kantorov]

Should it make more sense to first apply AvgPool(8, 8) and then appl selfsim_fc?

[A. Jabri]

Yes, I think this would make more sense than applying the selfsim_fc on the full feature map, but I haven't had a chance to test this.