Train PPT but got Loss = nan
isunLt opened this issue · comments
Thanks for sharing the code!
We follow the readme to prepare the enviorment and run the command:
sh scripts/train.sh -g 4 -d nuscenes -c semseg-ppt-v1m1-0-nu-sk-wa-spunet -n semseg-ppt-v1m1-0-nu-sk-wa-spunet
However, the loss turn to Nan in the middle of the training. Could the authors kindly help me with this problem?
Try again or set amp False. It does occur from time to time. Maybe clip loss within a range can solve it but I have not try it.
Try again or set amp False. It does occur from time to time. Maybe clip loss within a range can solve it but I have not try it.
Thank you for your timely responce. I have tried twice but all get loss = nan during the training. I will try to set amp False.
Try again or set amp False. It does occur from time to time. Maybe clip loss within a range can solve it but I have not try it.
I have tried semseg-ppt-v1m2-0-nu-sk-wa-spunet to use the decoupled segmentation head to train the network. It successfully runs through the training procedure but fails during the testing procedure as shown in the Fig. Can you help me with this problem so that I can successfully run the testing procedure?
