Training interruption
571129857 opened this issue · comments
Zhen Han commented
problem:
When using 4*1080Ti , training gets stuck, memory is not released, and no error is reported. Forced task termination, retraining is required
environment:
python 3.6
pytorch 1.1
cuda 10.0
code:
branch-1.1.0
haoxurt commented
I also encountered this problem, and I found it would arise at fixed interval if I continue train.