The loss always be inf

Question

The loss always be inf

Dinghow opened this issue 5 years ago · comments

I trained with the data generated by your tool TextRecognitionDataGenerator.
And trained with 100 iterations, the loss is always be inf, I'm wondering the reason, thx a lot.

Edouard Belval · Answer 1 · Tue Jan 22 2019 21:13:13 GMT+0800 (China Standard Time)

Hi, inf loss does happen from time to time but usually, it's either a random occurrence (so restarting the training will fix it) or a problem with the dataset.

How big is your dataset?
Are you training for Chinese? If so, what is the length of your CHAR_VECTOR?

Thank you

Dinghow Yang · Answer 2 · Tue Jan 22 2019 21:21:02 GMT+0800 (China Standard Time)

Thanks for your immediate reply. Tried to restart the training many times, without any improvement.
BTW, the size of my training dataset is 8000 in english. I just added some punctuations like ,.;'" to the CHAR_VECTOR.

Edouard Belval · Answer 3 · Tue Jan 22 2019 21:36:11 GMT+0800 (China Standard Time)

Interesting, I'll do some testing from my side see if I can get a test case.

If you could compress your 8000 samples dataset and send it to me to ensure reproducibility I would be grateful.

You can drop the file here: https://nextcloud.belval.org/index.php/s/MQsQrK8egatQtLo

Dinghow Yang · Answer 4 · Tue Jan 22 2019 21:46:43 GMT+0800 (China Standard Time)

Sure, I'm glad to do that, the size is 10001 actually. And my CHAR_VECTOR is
CHAR_VECTOR = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ-'.!?,\"" just reuse yours.

Edouard Belval · Answer 5 · Tue Jan 29 2019 11:48:05 GMT+0800 (China Standard Time)

Just a heads up to say that I was able to reproduce the issue. I'm quite busy at the moment but I'll try to troubleshoot it ASAP.

Thank you for bringing this to my attention.

Dinghow Yang · Answer 6 · Tue Jan 29 2019 14:23:41 GMT+0800 (China Standard Time)

OK, thx for your attention, hope for your update.

DamingN · Answer 7 · Mon Mar 25 2019 17:31:56 GMT+0800 (China Standard Time)

I face the same problem. Have you solved it?

DamingN · Answer 8 · Mon Mar 25 2019 17:37:02 GMT+0800 (China Standard Time)

My dataset is Chinese and the length of CHAR_VECTOR is 70000+, the long lengths can cause problems?

DamingN · Answer 9 · Mon Mar 25 2019 18:08:41 GMT+0800 (China Standard Time)

My dataset is Chinese and the length of CHAR_VECTOR is 70000+, the long lengths can cause problems?

sorry, my CHAR_VECTOR is 7000+.

The training process is too slow. How can I solve it? It trains five steps in four days.

Dinghow Yang · Answer 10 · Mon Mar 25 2019 22:11:05 GMT+0800 (China Standard Time)

I face the same problem. Have you solved it?

Not yet. I used this project without this problem https://github.com/wushilian/STN_CNN_LSTM_CTC_TensorFlow/blob/master/utils.py

DamingN · Answer 11 · Mon Mar 25 2019 22:14:31 GMT+0800 (China Standard Time)

I face the same problem. Have you solved it?

Not yet. I used this project without this problem https://github.com/wushilian/STN_CNN_LSTM_CTC_TensorFlow/blob/master/utils.py

Thank you. I'll try.