Feature sequences

[Kyongjun Kim]

Hello, I'm confused about token size because it's square(8x8 or 16x16) on ViT.

As I understand, feature sequence means 256 tokens with 1024 dimension vector.
And $N/N_w(=256/16)$ window feature sequences of Window Block means that there're 16 partitions with 16 sequences per partition.

Then 4 partitions of this figure is just for simplification? Or did I get it wrong?

[Zhigang Jiang]

Yes, for simplicity, this figure shows the case of $N_w=N/4$.

[Kyongjun Kim]

@zhigangjiang Thanks for your reply!