Backup broken after update to 64
schlypel opened this issue · comments
After updating there seems to be an additional ' in the command.
I had no time to look into the reasons yet, but here is some log:
ERROR 50 list
. Giving up after 5 attempts. BackendException: Error running 'rsync -e 'ssh -p 22 -oBatchMode=yes '-i /root/.ssh/id_rsa_miab -p 22'' '-e "/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab"' miab@X.X.X.X:/home/miab/encrypted/': returned 14, with output:
.
. rsync: [sender] Failed to exec /usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab: No such file or directory (2)
. rsync error: error in IPC code (code 14) at pipe.c(85) [sender=3.2.7]
. rsync: connection unexpectedly closed (0 bytes received so far) [sender]
. rsync error: error in IPC code (code 14) at io.c(231) [sender=3.2.7]
Has anybody else had this problem?
Same here, backup status email reports
duplicity: error: argument --gpg-options: expected one argument Something is wrong with the backup:
There is https://discourse.mailinabox.email/t/backup-script-failing/10734/62 which seems related.
The error is not the same but they talk about issues with the backup script and single quotes.
(I had a message similar to that of amazzi44; gpg: Note: '--decrypt' is not considered an option)
The cause for that is probably a966913 (Fix command line arguments for duplicity 2.1).
I updated my mailinabox, but still had duplicity 1.2.3 (I thought the setup script updates system packages, but I might be wrong).
Updating packages with apt gave me duplicity 2.1.1 and the backup.py is now working again.
duplicity --version
duplicity 2.1.1
I had my changes from the last problem reverted (having duplicity 1.2.3 build manually). Now my system is using duplicity 2.1.1, but the problem appears to result from an additional ' char after the parsed in port number.
Attempt of list Nr. 1 failed. BackendException: Error running
'rsync -e 'ssh -p 22 -oBatchMode=yes '-i /root/.ssh/id_rsa_miab -p 22'' <<< HERE [...]
I was able to use duplicity manually by running management/backup.py --verify which gives you the full call it uses.
From there I adjusted the call parameters to do an incremental backup and started it manually.
So neither the duplicity version nor the ssh or gpg options are the problem here.
I am pretty sure it comes from the part where the port number gets extracted from the target and added as ssh's -p option. I tried with a target that has no port (thus 22 being added as default) and explicitly port 22 set.
I had a look at management/backup.py and think the problem might come from somewhere around line 231.
But I am no python developer, so my debugging abilities are rather limited here.
Hi, I have a similar issue since v64.
When I run mailinabox/management/backup.py manualy I get:
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: Note: '--decrypt' is not considered an option
gpg: WARNING: no command supplied. Trying to guess what you mean ...
usage: gpg [options] [filename]
===== End GnuPG log =====
Traceback (most recent call last):
File "/root/mailinabox/management/backup.py", line 647, in <module>
perform_backup(full_backup)
File "/root/mailinabox/management/backup.py", line 318, in perform_backup
shell('check_call', [
File "/root/mailinabox/management/utils.py", line 126, in shell
ret = getattr(subprocess, method)(cmd_args, **kwargs)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'incr', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', "'--cipher-algo=AES256'", '--allow-source-mismatch', '/home/user-data', 'b2://***:***']' returned non-zero exit status 31.
@schlypel, I have the exact same error. Here's what I already tried:
- Upgraded to v64
- Removed the duplicity package (full purge)
- Reboot
- Installed duplicity 2.1.1-ppa202309021553~ubuntu22.04.1
- Reboot
- Backup still fails
- Downgraded duplicity to 0.8.21-1build1
- Reboot
- Back up still fails
- Upgraded duplicity to 2.1.1-ppa202309021553~ubuntu22.04.1
- Confirmed backup.py matches latest
Not sure what else to try - last successful backup was 8 days ago.
@britannic the duplicity downgrade was only necessary before the v64, the backup scripts have since been adjusted to work with duplicity v2.1.1.
I had solved this downgrade via manual build of duplicity v1.2.3 after the breaking changes. But since the v64 should not any longer require this, i had removed my own build, so the system was falling back to duplicity v2.1.1.
Also uninstalling and reinstalling duplicity v2.1.1 did not help me either and running mailinabox to set the system to a known good state did not result in working backups.
For safety measures I was able to manually run the backup. I had a look at the call to duplicity the management/backup.py would run and did this successfully.
So backing up works, just that there is a ' in the generated call, that should not be there.
@JvdBBBB , @ckeschnat and @julesfarrow I think the gpg related error might be something else.
Could be related to the way the backup script generates the call to duplicity, but is not the error I see.
I manually ran backup.py with the --full switch and got the error below:
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: Note: '--force-mdc' is not considered an option
gpg: Note: '--symmetric' is not considered an option
gpg: WARNING: no command supplied. Trying to guess what you mean ...
usage: gpg [options] [filename]
===== End GnuPG log =====
Traceback (most recent call last):
File "/root/mailinabox/management/backup.py", line 647, in <module>
perform_backup(full_backup)
File "/root/mailinabox/management/backup.py", line 318, in perform_backup
shell('check_call', [
File "/root/mailinabox/management/utils.py", line 126, in shell
ret = getattr(subprocess, method)(cmd_args, **kwargs)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'full', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', "'--cipher-algo=AES256'", '--allow-source-mismatch', '/home/user-data', 'file:///home/user-data/backup/encrypted']' returned non-zero exit status 31.```
@schlypel, I was able to run the the backup manually from the command line, building the options like you did and it worked fine. It seems there is still a bug in the way backup.py is calling duplicity.
I too have lost ability to perform any form of backup.
Updated to v64.
Duplicity 2.1.1 is installed.
management/backup.py Output =
usage: duplicity [-h] [--log-file log_filename]
[--cf-backend pyrax|cloudfiles] [--tempdir path]
[--s3-kms-grant s3_kms_grant] [--gpg-binary path]
[--volsize number] [--dry-run] [--no-print-statistics]
[--path-to-restore path] [--copy-blocksize number]
[--s3-unencrypted-connection] [--azure-max-block-size number]
[--mp-segment-size number] [--s3-multipart-chunk-size number]
[--sign-key gpg-key-id] [--s3-use-ia] [--progress]
[--par2-options options] [--hidden-encrypt-key gpg-key-id]
[--s3-use-glacier-ir] [--file-prefix-jsonstat string]
[--ftp-regular] [--max-blocksize number] [--timeout seconds]
[--ssl-cacert-file file] [--exclude-device-files]
[--include shell_pattern] [--time-separator char]
[--webdav-headers string] [--file-prefix-archive string]
[--s3-region-name s3_region_name] [--compare-data]
[--ssh-askpass] [--encrypt-secret-keyring path]
[--allow-source-mismatch] [--exclude shell_pattern]
[--log-timestamp] [--skip-if-no-change]
[--exclude-older-than time] [--verbosity verb]
[--restore-time time] [--no-files-changed]
[--encrypt-sign-key gpg-key-id] [--include-regexp regex]
[--metadata-sync-mode {full,partial}] [--config-dir path]
[--ssl-cacert-path path] [--file-prefix string]
[--ssh-options options] [--exclude-other-filesystems]
[--exclude-if-present filename] [--exclude-filelist filename]
[--scp-command command] [--filter-regexp] [--ignore-errors]
[--s3-use-deep-archive]
[--asynchronous-upload ASYNC_CONCURRENCY]
[--no-restore-ownership] [--rsync-options options]
[--imap-full-address] [--exclude-regexp regex]
[--file-changed path] [--name backup name]
[--progress-rate number] [--ssl-no-check-certificate]
[--copy-links] [--log-fd file_descriptor] [--no-encryption]
[--azure-blob-tier Hot|Cool|Archive] [--ftp-passive]
[--filter-literal] [--show-changes-in-set number]
[--mf-purge] [--s3-use-rrs] [--backend-retry-delay seconds]
[--s3-kms-key-id s3_kms_key_id]
[--s3-use-server-side-kms-encryption] [--s3-use-onezone-ia]
[--idr-fakeroot path] [--force] [--encrypt-key gpg-key-id]
[--gpg-options options] [--filter-globbing] [--jsonstat]
[--include-filelist filename] [--par2-volumes number]
[--archive-dir path] [--files-from filename]
[--imap-mailbox imap_mailbox]
[--s3-multipart-max-procs number] [--s3-use-glacier]
[--par2-redundancy number] [--full-if-older-than time]
[--sftp-command command] [--b2-hide-files] [--use-agent]
[--filter-strictcase] [--num-retries number]
[--null-separator] [--rename from to from to]
[--swift-storage-policy policy] [--filter-ignorecase]
[--file-prefix-manifest string] [--no-compression]
[--s3-endpoint-url s3_endpoint_url]
[--azure-max-connections number] [--version]
[--numeric-owner] [--file-prefix-signature string]
[--azure-max-single-put-size number]
duplicity: error: argument --gpg-options: expected one argument
Something is wrong with the backup:
Which is all gobbledegook to me
Running on AWS EC2 I encountered a similar issue.
Ubuntu 22 arm64. This is my kernel: 6.2.0-1012-aws #12~22.04.1-Ubuntu SMP Thu Sep 7 16:00:15 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux.
I noticed that the apt duplicity package only goes to 0.8.21, so I removed it, and installed the snap instead, which is current at 2.1.1.
To make the snap work with backup.py, within the script I had to update the path to duplicity, since it defaults to /usr/bin/duplicity, but the snap lives at /snap/bin/duplicity.
Backups are working now, but this doesn't feel like the optimal solution. Looking forward to this being addressed in future releases.
Progress, somewhat...
On the plus side, the error I received initially (and circumvented by running an adapted backup script) can't be reached anymore...
On the minus, it's because I receive the duplicity error @britannic reported earlier, since I received an update for duplicity...
Traceback (most recent call last):
File "/usr/bin/duplicity", line 32, in <module>
from duplicity import gpg
File "/usr/local/lib/python3.10/dist-packages/duplicity/gpg.py", line 40, in <module>
from duplicity import tempdir
File "/usr/local/lib/python3.10/dist-packages/duplicity/tempdir.py", line 30, in <module>
from future import standard_library
ModuleNotFoundError: No module named 'future'
I think, I will do a clean install of MiaB and restore from Backup (if I manage to create a recent one...)
More progress...
the duplicity error stems from missing python3-future, which can easily be apt installed and fixes this error.
After installing I was able to use my own Backup script once again.
management/backup.py still times out it seems.
So, do manual backup and then do a reinstall and restore still looks most promising.
On the other hand, testing your backups is a good thing 😁
Would you mind sharing your backup script?
More progress... the duplicity error stems from missing python3-future, which can easily be apt installed and fixes this error. After installing I was able to use my own Backup script once again.
management/backup.pystill times out it seems. So, do manual backup and then do a reinstall and restore still looks most promising.On the other hand, testing your backups is a good thing 😁
@schlypel, so I already have python3-future and duplicity 2.1.3 installed:
pip3 install future Requirement already satisfied: future in /usr/local/lib/python3.10/dist-packages (0.18.3)
root@box:~# apt list duplicity Listing... Done duplicity/jammy,now 2.1.3-ppa202310081632~ubuntu22.04.1 amd64 [installed]
Manually running the generated command line from the backup.py error works just fine:
Error:
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'incr', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', "'--cipher-algo=AES256'", '--allow-source-mismatch', '/home/user-data', 'file:///home/user-data/backup/encrypted']' returned non-zero exit status 31.
Extracted command line (works without error):
root@box:~# /usr/bin/duplicity incr --verbosity warning --no-print-statistics --archive-dir /home/user-data/backup/cache --exclude /home/user-data/backup --volsize 250 --gpg-options "--cipher-algo=AES256" --allow-source-mismatch /home/user-data file:///home/user-data/backup/encrypted GnuPG passphrase for decryption: Retype passphrase for decryption to confirm:
IMHO it seems to be an issue with the backup.py generated command line.
Would you mind sharing your backup script?
It is just the extracted commandline and an export of the first line of the contents of your secret-key.txt (since gpg only reads the first line as key for its encryption).
#!/bin/bash
export PASSPHRASE=XXXXXXXXXXXXXXXXXXXXXXXXXX
duplicity --archive-dir /home/user-data/backup/cache --exclude /home/user-data/backup --volsize 250 --gpg-options '
--cipher-algo=AES256' --allow-source-mismatch --ssh-options='-i /root/.ssh/id_rsa_miab -p 22' --rsync-options='-e "
/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab"' /home/user-data rsync://
USER@SERVER:PORT//YOUR/PATH/encrypted
IMHO it seems to be an issue with the backup.py generated command line.
Yes, that was my result too.
Only thing that I find interesting, from all the users that are relying on miab, we are just a small group, that apparently has this problem, although the server configurations should be mostly identical for everyone.
So I still think there is a high probability, that the wrongly generated command (I saw the problem in the order of ' and " in the gpg/rsync/ssh options) stems from a specific thing in our backup target path.
On the other hand, I use rsync mode, some of you seem to use local paths, maybe the majority uses S3 and doesn't have this problem...
Still, making a backup (manually) and reinstalling miab seems to be the way to go.
@schlypel, I also tried the backup with B2 (Backblaze) storage and got the same error.
I got some help from SHAdmin on the MIAB Forum here.
I did the following:
rm -rf /usr/local/lib/python3.10/dist-packages/duplicity/apt-get purge duplicityapt-get reinstall python3apt-get reinstall duplicity
After this, I was able to run backup.py without errors.
@britannic that actually fixed it for me too.
So apparently it did not have anything to do with the 's
I hope this fixes it for the others too and close the issue for now.
Thanks
Hey everyone 👋
I found this issue because I'm running Mail-in-a-Box on Ubuntu 22 on a Raspberry Pi 4, and the newest version of duplicity I can pull down via the Jammy PPA seems to be 0.8.21-1build1.
Operating System: Ubuntu 22.04.3 LTS
Kernel: Linux 5.15.0-1045-raspi
Architecture: arm64Obviously, that's not a miab responsibility, but here's my quick workaround to get things backed up:
sudo apt purge duplicity && sudo apt autoremove -y
sudo snap install duplicity --classic
sudo ln -s /snap/bin/duplicity /usr/bin