Resume-from-failed functionality
rusq opened this issue · comments
After 17GB of export, my slackdump process died with a 500 response from Slack. I'd love to be able to start again where it left off. It was a 500 error, so hopefully it was resolved on their end (and not due to bad data etc).
2022/12/06 21:32:28 application error: export error: channels: error: failed to dump "dataeng-alerts-high" (CM0BW29LZ): callback error: failed to dump channel:thread CM0BW29LZ:1575031520.121400: slack server error: 500 Internal Server Error
Being able to resume where it died would be huge for someone trying to get a complete workspace export. And it's probably as easy as enumerating everything (into a file) on process start, and removing lines from the file as tasks get completed. On start up, it can see if this file already exists and start with the next line as the first task.
Thoughts?
edit: Thinking about it a bit more - an "if this file already exists, skip it" check before exporting a channel/thread/file might help speed things up dramatically and accomplish almost the same thing. Hmm
Hey @G-Harmon, yeah, I'm planning to combine those two in v3.
I haven't decided on the CLI flags yet, but the general idea is the following:
- once the "resume" or "continue" flag is given along with the existing ZIP archive or directory; do
- If the directory is empty - STOP
- Detect the type
- Read the existing files and detect the latest message time for each of the channels
- Start processing each channel from the respective timestamp.
Meanwhile, you could check the shell script that levygroker has contributed recently: https://github.com/rusq/slackdump/tree/master/contrib/incremental_backup
(You can read the whole conversation in #193 )
It's been 3 days that I was trying to export the workspace.
Single disconnect.
And you start it over :pain: