New training gets killed after some time and never finish/converge?

Question

New training gets killed after some time and never finish/converge?

kasunsp opened this issue 5 years ago · comments

I could generate the results in the paper.
but when I try training the model from the scratch, then the program always get killed at
2019-02-20 07:53:55,385 - [INFO] - E:0 Train Accuracy (27232/292459): 93.46 975.36 new_run 0.0
Killed
I run with GPU and without GPU but nothing changed.
How can I run the model training from the scratch?
Thanks,

Shikhar Vashishth · Answer 1 · Thu Feb 21 2019 12:14:30 GMT+0800 (China Standard Time)

Hi @kasunsp,
This looks like an issue with your machine. Anyways, I will check everything once again and let you know.

Thanks

Rishabh Joshi · Answer 2 · Fri Mar 08 2019 14:53:54 GMT+0800 (China Standard Time)

Do check if you have sufficient RAM. You can check this by running htop (or even the System monitor) and then starting the code.

Shikhar Vashishth · Answer 3 · Sat Mar 09 2019 12:15:38 GMT+0800 (China Standard Time)

Hi @kasunsp,
I am not facing this issue with the code. Seems like there is some issue in your system only.