Question
song-cc opened this issue · comments
When I run train.py, I found the loss is 'nan' at behind of 2-epochs. Do you have this problem when you train. So I want to know why is it and how to solve this problem.
Duplicate of #
Hi,
I have not encountered this problem before. Did you check if the data-sets are normalized to -1 to 1 and being passed to the models properly?
You can check the npz file by opening and then printing one row of the values.
It can be also a problem of the version of tf/keras.
Just to be sure, I will try to run the train.py again and report you the output here.
Thanks
I am sure that my tensroflow and keras version are 2.0.0 and 2.3.1 . And I am sure that the value of .npz file is between -1 and 1. But I will still got the same problem.
Your problem is a classic case of gradient explosion. There are tons of github issues/stackexchange discussion on this.
Most of them mention the dataset being normalized (-1.0 to 1.0 in our case) properly and made into float values.
Can you mention which Data-set you used DRIVE, CHASE or STARE. Also, can you provide the .npz file as bitbucket, dropbox or drive link here ? I want to try it on my code.
Thanks
I use DRIVE dataset,but the .npz is large.
Could you Email me? I will Email the .npz file.
My Email is songchongchongde@163.com
Thanks very much.
If you can not download from:https://pan.baidu.com/s/1I8CNRCq_1WZ-TrAO83Nyzg
the code is :driv ,please email me
Please use this npz file and train using the python file given in the repository. Also post the results here.
Thanks
https://drive.google.com/file/d/1j1BErDnxJIjA3VEgrr66tXQFyW-NTzDr/view?usp=sharing
Thanks very much.
It's better if you can link the original DRIVE data images.
I want to known why we got this problem.
It's already given in the Readme.MD file.