How to determine the value of fake_class_label in models_mage.py

Question

How to determine the value of fake_class_label in models_mage.py

xcyuyuyu opened this issue a year ago · comments

Tianhong Li · Answer 1 · Tue Mar 21 2023 12:06:22 GMT+0800 (China Standard Time)

It can be of any value other than the tokenzier codebook indices range (0-1023).

xcyuyuyu · Answer 2 · Wed Mar 22 2023 11:36:45 GMT+0800 (China Standard Time)

Thanks. BTW, when I do the pretrain with 8 V100 with 512batchsize, why does the loss reduce to 2.6 and no longer reduce, is it really needs a large batchsize for training?
I trained on the custome dataset with 1.4w images

Tianhong Li · Answer 3 · Wed Mar 22 2023 11:40:21 GMT+0800 (China Standard Time)

A loss of 2.6 is already quite low if you use the default masking strategy: in our experiment, the training loss converges around 5.7-5.8.