Nan loss during training

Question

Nan loss during training

Waynechen92 opened this issue 2 years ago · comments

Has anyone ever met Nan lossing during training for bts_main?
Any possible solution for that?

Xuanlong Yu · Answer 1 · Wed Nov 23 2022 18:27:00 GMT+0800 (China Standard Time)

Hi, thank you for leaving the comments.
Actually, it appears in the original bts training as well, you can check here to find some suggestions given by the authors of bts.

Waynechen92 · Answer 2 · Thu Nov 24 2022 13:26:58 GMT+0800 (China Standard Time)

Thanks for your reply xuanlong, I'm also wondering whether it is correct to have negative loss in training for bts_main

Xuanlong Yu · Answer 3 · Sat Nov 26 2022 04:34:42 GMT+0800 (China Standard Time)

Several common loss functions are indeed non-negative such as MSELoss or L1Loss. But it is fine to have a negative loss value during training, not only in LDU case. You can check here for example.