Giters

junchen14 / LoMaR

LoMaR (Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

nan loss

jimmy-dq opened this issue · comments

commented

Hi junchen,
Thanks for your great work. Recently when I try your local reconstruction training, I found the NaN loss is easily occurred during the training. Any suggestions for this? Thanks.

commented

Hi can you share more about your training details so I can better understand the problem. Actually I seldom face nan problem in my experiments.

commented

Hi can you share more about your training details so I can better understand the problem. Actually I seldom face nan problem in my experiments.

Thanks for your reply. I disable the iRPE and use my own dataset for training. The nan loss occurs at the beginning (i.e., epoch-8). This problem is solved when I disable the AMP training, i.e. amp.autocast(enabled=False).