Hourly Backups & Exports to Google Drive seem to fail
nucleare opened this issue · comments
Even though I've successfully connected my Google Drive and successfully uploaded my workspace.zip file
Mounted at /content/drive
/content
Done!
During training, the automatic backup seems to fail.
[04:14:41][#000785][9566ms][1.6624][1.9476]
[04:29:50][#000861][9324ms][1.6438][1.9358]
cp: failed to access '/content/drive/My Drive/': Transport endpoint is not connected
Backed up!
[04:46:43][#000934][3614ms][1.6339][1.9001]
I have tried modifying the code for training and added the force_remount tag to the training code but it still produces the same error during automatic hourly backup.
If I stop training and manually try to export my model folders and in the input field tried to rename the entry to "/workspacemanualback" I receive this error:
Mounted at /content/drive
/content/workspace
cp: cannot stat '/workspacemanualback': No such file or directory
rm: cannot remove '/workspacemanualback': No such file or directory
Done!
To see if renaming it caused an issue, i changed it back to workspace by default and the error is still receive the same error:
Mounted at /content/drive
/content/workspace
cp: cannot stat 'workspace': No such file or directory
rm: cannot remove 'workspace': No such file or directory
Done!
Although I have already cleared it from a previous session that failed and cant copy all the other errors in all it's other attempts to save or backup. this is what was shown for Get Result video
python3: can't open file 'DeepFaceLab/main.py': [Errno 2] No such file or directory
cp: cannot stat '/content/workspace/result.mp4': No such file or directory
Basically, I have yet to be able to use any session of Google Colab to help me with training but it's worth noting that I am only uploading my workspace AFTER i've done the extraction and some masking for Xseg in advance, which means my workspace.zip is 10gb+ while the src and dst video files are 2gb and 6gb respectively
I have updated to Colab Pro and still cant seem to export any work that is done on Colab, any ideas?
Some notes that may or may not help:
- I've also tried connecting to Google Drive using not only the form built in, but also the icon the left.
- I copied the notebook to my own drive before executing any cells
- I've also tried manually uploading all the files by upload and not using Google drive but couldnt figure out how to download them
Is there any other ways to pull/download the data off the notebook aside from exporting to Google Drive that you know if?
The issue was discovered and was particular to the environment, the code from this repository has been checked and is unaffected and therefore does not represent a universal bug. Somehow the coding for the mount director in Google drive was changed in the Training code side instructions.