how to perform correlation for 4d tensor.

[aiwithshekhar]

thanks for the amazing work!!

For my work the projector outputs two tensors of size [128, 3, 64, 64], how can i find the correlation matrix between these.
One way is to to flatten it in [128, -1] format but it will loose the spatial information so i didn't wanted to do that is their any other way?

[Jure Zbontar]

With Barlow twins we would use average pooling to go from [128, 3, 64, 64] to [128, 3]. We would then compute the [3, 3] cross-correlation matrix and compute the loss from it (we used more than 3 features, though).

You could reshape your tensor form [128, 3, 64, 64] to [128, 12288]. The size of the cross correlation matrix would then be [12288, 12288]. I don't think this would loose spatial information.

There is also an in-between approach where you would use adaptive average pooling to transform [128, 3, 64, 64] to, say, [128, 3, 8, 8] and then reshape to [128, 192]. We tried something like that, but it didn't improve much over the first approach.