trainning loss=NAN
Adarine99 opened this issue · comments
Hi,
I'm sorry to disturb you again，but I have no clue how to solve this.Testing on my own data,I use one human segmentation image and origin images as input, then run the demo.py , got another problem as follows:
Started loading
Performing the data augmentation
Done initializing Dataset
Initializing from specified pre-trained model...
INFO:tensorflow:Restoring parameters from models/OSVOS_parent/OSVOS_parent.ckpt-50000
Weights initialized
Start training
2018-09-01 15:43:31.268901 Iter 10: Training Loss = 232345665536.0000
2018-09-01 15:45:39.789833 Iter 20: Training Loss = nan
2018-09-01 15:47:21.016101 Iter 30: Training Loss = nan
2018-09-01 15:49:22.912398 Iter 40: Training Loss = nan
2018-09-01 15:51:17.401571 Iter 50: Training Loss = nan
Finally,Igot the output images,and the result is terrible.
Is there something wrong?Is there anyone can help me?
Thank you so much!
Hi, it could be one of the following things:
- Learning rate is too high and should be reduced (default is 1e-8, please try 1e-9 or 1e-10).
- Your label is not binary in case you don't use the label preprocessing of this code. Please make sure your label only has values of {0, 1}.
I hope this helps.
Thank you , my problem is No. 1( worked with 1e-9). :)