Handle crashed runs on wandb
euanong opened this issue · comments
When you KeyboardInterrupt, wandb seems to upload some extra data... note-to-self to do something about this.
Related: if we checkpoint the model so much that the data exceeds ~115 MB, the uploader seems to hang forever after ~115 MB, I wonder if there's a way to manually resume