Cloud backup only changed games
davidmartos96 opened this issue · comments
What's your idea?
Hello! Loving Ludusavi, great job!
One thing I would really appreciate to see is to create a backup and upload only the games that have been changed. This "changed games" feature appears to exist already, as the UI has a different icon for these entries.
This would greatly help performance in the creation of the backups when the total size is quite big.
Just for context, the backup size on my case is about 1.4 GB, but even though the preview takes just a couple seconds, the backup command or UI button takes about 5 minutes, before even showing the "Scan" step, on an SSD. Is there a special previous step before the Scan phase that could take that much time? In part is a bit of a UX issue, because neither the UI, nor the CLI show any progress, just frozen until several minutes later.
If that is an issue, the feature request might not be as important, but if not, it could be very useful to reduce the amount of work.
Thank you for considering!
Is there a special previous step before the
Scanphase that could take that much time?
Yeah, if you have synchronize automatically enabled, then there's a cloud phase before and after the scan phase. It's supposed to show a progress bar during those phases, but the progress is based on parsing the output from Rclone, so something might be going wrong there. Some ideas:
- What version of Rclone do you have?
- If you go to other -> cloud -> upload/download -> preview, does it have the same issue?
- In the cloud config, if you specify
--transfers=1 --fast-list --ignore-checksumfor the Rclone arguments, does that help?
One thing I would really appreciate to see is to create a backup and upload only the games that have been changed.
For what it's worth, this is how it works now:
- If you're doing a full scan:
- The pre-scan cloud phase will verify that the entire backup folder is in sync
- The post-scan cloud phase will try to upload everything and let Rclone figure out what doesn't need to be changed
- If you're scanning a single game:
- The pre-scan cloud phase will verify only that game
- The post-scan cloud phase will upload only that game
The full post-scan cloud phase could be limited to the changed games only, since we know from the pre-scan cloud phase that everything else is in sync. However, for the full pre-scan cloud phase, it does need to check everything, since the cloud backups might contain some new games.
We could also add an option to each game's ... menu to upload/download just that game.
@mtkennerly Thanks for the detailed response. Nice coincidence, I actually did some experiments with this earlier today. I discovered the --fast-list and that flag alone made the initial cloud phase much faster (10 s vs 2 min) (Using Google Drive)
The rclone version I'm using is v1.64.2 and preview is not slow. It was a combination of the 2 cloud phases.
First cloud phase is "solved" with a proper usage of rclone with fast-list. The post-scan cloud phase is still quite slow though, since it's reuploading everything. One thing I noticed while debugging is that the changes vector obtained in the pre-scan cloud phase includes everything, even unchanged games. This is because the mapping.yml files has a different when property (Image below). So even if the checksum is the same, it is flagged as changed. Is this expected, or should those be filtered out in some way? As it behaves now the "Cloud conflict" error message is very likely to appear, because most of the times the vector will not be empty.
P.S Out of curiosity, do you know if there is any reason fast-list is not true by default? I haven't found anything online.
The post-scan cloud phase is still quite slow though, since it's reuploading everything.
I'll try limiting the upload to just the games with changes 👍
One thing I noticed while debugging is that the
changesvector obtained in the pre-scan cloud phase includes everything, even unchanged games. This is because the mapping.yml files has a differentwhenproperty (Image below). So even if the checksum is the same, it is flagged as changed. Is this expected, or should those be filtered out in some way?
If I understand correctly, your local and cloud backups are identical except for the timestamp? How did that happen? Ludusavi shouldn't create a new backup if the saves are the same. But if you do have different timestamps in the local and cloud backups, then the conflict error is expected.
P.S Out of curiosity, do you know if there is any reason fast-list is not true by default? I haven't found anything online.
I'm not sure. The Rclone help describes it as: Use recursive list if available; uses more memory but fewer transactions , so I guess the Rclone developers prefer to prioritize low memory usage over speed. I'm open to making it a default in Ludusavi.
How did that happen
Thinking about it, it must have been a manual cloud upload of the backup folder I did a few months ago.
Glad to hear ludusavi is not supposed to upload in those cases.
Having fast-list as a default and have an option to opt out would be nice indeed. I'm sure there are some people that just don't know about the flag and can afford having it on.
Looking forward to the upload optimization. Thanks for considering!
I've just updated the master branch for the upload optimization. Could you give it a try and let me know if it helps? If you need a prebuilt version, just let me know which OS and I can provide one.
@mtkennerly It's much better now. Thanks a lot!
Related to the original post, it might be useful to log the current step in stdout, in addition to the logger. It would improve UX, because the user would know what kind of work is being done.
The Scan with the progress bar is great in the CLI, but for the pre-scan and post scan syncs nothing is shown in the terminal. A simple, indeterminate and static progress message could work: Something like "Checking cloud sync before backup..." and "Performing cloud sync for {N} changed games...",
The CLI was supposed to show a cloud progress bar, but it didn't display because of #222. I just fixed that, so you should see the progress bar and timer now :)
That said, the cloud progress bar may not actually move, depending on the output we get from Rclone. In my testing with --fast-list, we don't get enough data to move the bar in the pre-scan phase, but we do in the post-scan phase once Rclone has discovered the differences and starts uploading.
@mtkennerly Thank you!