got NAN after trainging some iters
harlem867 opened this issue · comments
@motokimura dear motokimura,
when I use your code to train COCＯ dataset, the loss declined normally and suddenly xy and wh loss changed to nan after about 10w iters. The learning rate is 0.001 during the whole training process and i didn't change your default settings.
When I trained on my own dataset, got the same problem.
Can you give me some advice? Thank you.
Hi, can you try with smaller learning rate?
Making GRADIENT_CLIP smaller might help also.
Or, it might help to add constant offsets to predicted uncertainties as done in official implementation. For details, please see jwchoi384/Gaussian_YOLOv3#40.