Giters

wayveai / fiery

PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"

Home Page:https://wayve.ai/blog/fiery-future-instance-prediction-birds-eye-view

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

NAN loss after one epoch

sjbling opened this issue · comments

commented

HI,
I train the model based on nuscenes with baseline.yml, but I got the nan loss after one epoch, how to train the model from scratch? Looking forward to your reply！

commented

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

I also tried the same setup and have the same issue. I tried to dig a little bit deeper and found out that running means and running variances of batch normalization are getting first to -infinity and then to nan. This also causes the loss to go to nan. The loss seems to get back from the nan-values, however, the model will still output -infinities for the segmentation.

Best regards,
Markus

commented

Hey both of you, and sorry for the late answer. It is a known issue, and it seems like training the whole network from scratch leads to instability. The fix is to load pre-trained weight from a 1-timestep model (FIERY Static) first because training the whole future prediction model as discussed here: #8

commented

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

After I train the model for epoch I got loss as negative value.Do you know the reason?

Best regards,
Cara

commented

Hey Cara, the negative loss value is due to the adaptive weighting of the losses using uncertainty. There is an additional loss term that prevents the weight from growing too large (see page 5 https://arxiv.org/pdf/1705.07115.pdf)