How to stop and resume training
Traincraft101 opened this issue · comments
Do you have to do anything specific to stop the training if you want to test the model? can I just close the command prompt? How do you resume training when you stop it?
To stop usually I just kill .
To restore you have to use the argument --restore_from and appoint the last version of yout model at the logdir folder.
I tried this, but I am seeing no indication that it worked. The step number reset back at 0 and the loss shot back up to early training levels.