Pre-train ViT on unsupervised task?

[Neel Gupta]

Thanks a lot for this repo! I am looking to pre-train ViT in an unsupervised way, but it doesn't seem like it's possible with this repo.

Is there any way I could do that? would you happen to know any other method/repo where we can pre-train it in an unsupervised manner (i.e on a dataset not labeled)?

do you think that specifying a dummy class and pre-training that way may provide extra accuracy when fine-tuning on the labeled dataset?

Many thanks 🤗

[YuanLi]

Hi, if you want to know more about self-supervised learning by vision transformer, I recommend this paper