Fail to backup to a synology btrfs drive
foux opened this issue · comments
Hi there!
I'm trying to backup from my local workstation to a synology formated drive.
Here is my btrbk conf :
transaction_log /var/log/btrbk.log
lockfile /var/lock/btrbk.lock
snapshot_dir snapshots
incremental yes
snapshot_preserve_min 1d
snapshot_preserve 24h 3d
target_preserve_min no
target_preserve 24h 7d 4w 12m
timestamp_format long
preserve_hour_of_day 12
preserve_day_of_week monday
ssh_identity /root/.ssh/id_rsa
snapshot_create onchange
btrfs_commit_delete each
# Root
volume /mnt/real_root
target send-receive /datas/samsung_backups
target send-receive /datas/backups/vili
target send-receive ssh://192.168.10.11/volume1/backups/vili
target_preserve 24h 7d 4w
subvolume @
subvolume @/opt/docker
A manual btrfs send-receive from one of my local snapshots to the synology, works like a charm, ie :
btrfs send /mnt/real_root/snapshots/@.20210420T0800 | ssh root@ds1815.local "btrfs receive /volume1/backups/vili"
But when dry-runing btrbk I've got the following error :
btrbk command line client, version 0.29.1 (Tue Apr 20 09:01:29 2021)
Using configuration: /etc/btrbk/btrbk.conf
Ignoring non-parseable btrfs mountpoint on ssh://192.168.10.11: "/volume1/docker/#snapshot"
WARNING: Skipping target "192.168.10.11:/volume1/backups/vili": Failed to fetch subvolume detail
WARNING: ... Command execution failed (exitcode=1)
WARNING: ... sh: ssh -i /root/.ssh/id_rsa root@192.168.10.11 'btrfs subvolume show /opt'
WARNING: ... not a subvolume: /opt
WARNING: ... Failed to get subvol info /opt: 1
WARNING: ...
Ignoring non-parseable btrfs mountpoint on ssh://192.168.10.11: "/volume1/docker/#snapshot"
WARNING: Skipping target "192.168.10.11:/volume1/backups/vili": Failed to fetch subvolume detail
WARNING: ... Command execution failed (exitcode=1)
WARNING: ... sh: ssh -i /root/.ssh/id_rsa root@192.168.10.11 'btrfs subvolume show /opt'
WARNING: ... not a subvolume: /opt
WARNING: ... Failed to get subvol info /opt: 1
WARNING: ...
Assuming non-present subvolume "@.20210420T0901" in skipped targets: "192.168.10.11:/volume1/backups/vili"
Creating subvolume snapshot for: /mnt/real_root/@
[snapshot] source: /mnt/real_root/@
[snapshot] target: /mnt/real_root/snapshots/@.20210420T0901
Assuming non-present subvolume "docker.20210420T0901" in skipped targets: "192.168.10.11:/volume1/backups/vili"
Creating subvolume snapshot for: /mnt/real_root/@/opt/docker
[snapshot] source: /mnt/real_root/@/opt/docker
[snapshot] target: /mnt/real_root/snapshots/docker.20210420T0901
Checking for missing backups of subvolume "/mnt/real_root/@" in "/datas/samsung_backups/"
No missing backups found
Checking for missing backups of subvolume "/mnt/real_root/@" in "/datas/backups/vili/"
No missing backups found
Checking for missing backups of subvolume "/mnt/real_root/@/opt/docker" in "/datas/samsung_backups/"
No missing backups found
Checking for missing backups of subvolume "/mnt/real_root/@/opt/docker" in "/datas/backups/vili/"
No missing backups found
Cleaning backups of subvolume "/mnt/real_root/@": /datas/samsung_backups/@.*
Deleted 0 subvolumes in: /datas/samsung_backups/@.*
Cleaning backups of subvolume "/mnt/real_root/@": /datas/backups/vili/@.*
Deleted 0 subvolumes in: /datas/backups/vili/@.*
WARNING: Skipping cleanup of snapshots for subvolume "/mnt/real_root/@", as at least one target aborted earlier
Cleaning backups of subvolume "/mnt/real_root/@/opt/docker": /datas/samsung_backups/docker.*
Deleted 0 subvolumes in: /datas/samsung_backups/docker.*
Cleaning backups of subvolume "/mnt/real_root/@/opt/docker": /datas/backups/vili/docker.*
Deleted 0 subvolumes in: /datas/backups/vili/docker.*
WARNING: Skipping cleanup of snapshots for subvolume "/mnt/real_root/@/opt/docker", as at least one target aborted earlier
Completed within: 1s (Tue Apr 20 09:01:30 2021)
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.29.1)
Date: Tue Apr 20 09:01:29 2021
Config: /etc/btrbk/btrbk.conf
Dryrun: YES
Legend:
=== up-to-date subvolume (source snapshot)
+++ created subvolume (source snapshot)
--- deleted subvolume
*** received subvolume (non-incremental)
>>> received subvolume (incremental)
--------------------------------------------------------------------------------
/mnt/real_root/@
+++ /mnt/real_root/snapshots/@.20210420T0901
!!! Target "192.168.10.11:/volume1/backups/vili" aborted: Failed to fetch subvolume detail
/mnt/real_root/@/opt/docker
+++ /mnt/real_root/snapshots/docker.20210420T0901
!!! Target "192.168.10.11:/volume1/backups/vili" aborted: Failed to fetch subvolume detail
NOTE: Some errors occurred, which may result in missing backups!
Please check warning and error messages above.
NOTE: Dryrun was active, none of the operations above were actually executed!
And indeed, /opt isn't a subvolume on the remote host, so it has no reason to work, but I can't figure out why btrbk tries to get the subvolume infos for /opt given that I'm not backing up there. Is there an issue with my config?
Thanks in advance for your help.
Doing the same when initiating the backup from the synology server results in the same error :
btrbk.conf :
transaction_log /volume1/backups/btrbk/btrbk.log
#stream_buffer 256m
snapshot_dir snapshots
snapshot_create always
incremental yes
preserve_hour_of_day 0
preserve_day_of_week sunday
#snapshot_preserve_min 1d
#snapshot_preserve <NN>h <NN>d <NN>w <NN>m <NN>y
#target_preserve_min no
#target_preserve <NN>h <NN>d <NN>w <NN>m <NN>y
# Specify SSH private key for "ssh://" volumes / targets:
ssh_identity /root/.ssh/id_rsa
ssh_user root
#ssh_compression no
#ssh_cipher_spec default
lockfile /volume1/backups/btrbk/btrbk.lock
btrfs_commit_delete each
# Resume backups from remote host which runs its own btrbk instance
# creating snapshots for "home" in "/mnt/btr_pool/btrbk_snapshots".
volume ssh://192.168.10.5/mnt/real_root
snapshot_create no
snapshot_preserve_min all
target /volume1/backups/vili
subvolume @
btrbk dryrun :
btrbk command line client, version 0.31.2 (Tue Apr 20 09:28:25 2021)
Using configuration: /volume1/backups/btrbk/btrbk.conf
Ignoring non-parseable btrfs mountpoint on LOCAL:: "/volume1/docker/#snapshot"
WARNING: Skipping target "/volume1/backups/vili": Failed to fetch subvolume detail
WARNING: ... Command execution failed (exitcode=1)
WARNING: ... sh: btrfs subvolume show /opt
WARNING: ... not a subvolume: /opt
WARNING: ... Failed to get subvol info /opt: 1
WARNING: ...
WARNING: Skipping cleanup of snapshots for subvolume "192.168.10.5:/mnt/real_root/@", as at least one target aborted earlier
Completed within: 1s (Tue Apr 20 09:28:26 2021)
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.31.2)
Date: Tue Apr 20 09:28:25 2021
Config: /volume1/backups/btrbk/btrbk.conf
Dryrun: YES
Legend:
=== up-to-date subvolume (source snapshot)
+++ created subvolume (source snapshot)
--- deleted subvolume
*** received subvolume (non-incremental)
>>> received subvolume (incremental)
--------------------------------------------------------------------------------
192.168.10.5:/mnt/real_root/@
!!! Target "/volume1/backups/vili" aborted: Failed to fetch subvolume detail
NOTE: Some errors occurred, which may result in missing backups!
Please check warning and error messages above.
NOTE: Dryrun was active, none of the operations above were actually executed!
Thanks for reporting!
It looks like some error with mountinfo in combination with docker and/or the mount point containing a hash # character (which is not allowed and thus ignored by btrbk): /volume1/docker/#snapshot.
I'm not able to reproduce this, could you please provide me more information (of both workstation and NAS):
- kernel version:
uname -a - btrfs-progs version:
btrfs --version - mountinfo:
cat /proc/self/mountinfo - most important, btrbk debug log:
btrbk run -n -l debug
Note that the logs may contain sensitive data, you can send them to me via email if you like: axel (at) tty0.ch
If found the issue : it was because of rbind mounts of BTRFS folders that weren't BTRFS subvolumes. Changing them to subvolumes fixed the issue.
I guess it shouldn't fail for that, but at least it's an easy fix.
But I ran into another issue because of synology... bad BTRFS. When syncing I got into the following issue :
[send/receive] checking target metadata: ds1815.local:/volume1/backups/vili/@.20210419T1649
ERROR: Failed to send/receive subvolume: /mnt/real_root/snapshots/@.20210419T1649 -> ds1815.local:/volume1/backups/vili/@.20210419T1649
ERROR: ... Command execution failed (exitcode=-1)
ERROR: ... sh: btrfs send /mnt/real_root/snapshots/@.20210419T1649 | ssh -i /root/.ssh/id_rsa root@ds1815.local 'btrfs receive /volume1/backups/vili/'
ERROR: ... attribute 12 requested but not present.
ERROR: ... lsetxattr var/log/journal ignore system.posix_acl_default=
ERROR: ... lsetxattr var/log/journal ignore system.posix_acl_access=
ERROR: ... lsetxattr var/log/journal/c1d7e0edcc6b8f630d41488e60739a07 ignore system.posix_acl_default=
ERROR: ... lsetxattr var/log/journal/c1d7e0edcc6b8f630d41488e60739a07 ignore system.posix_acl_access=
ERROR: ... lsetxattr var/log/journal/c1d7e0edcc6b8f630d41488e60739a07/system.journal ignore system.posix_acl_access=
ERROR: ... lsetxattr var/log/journal/c1d7e0edcc6b8f630d41488e60739a07/user-1026.journal ignore system.posix_acl_access=
ERROR: ... lsetxattr var/log/journal/c1d7e0edcc6b8f630d41488e60739a07/user-1000.journal ignore system.posix_acl_access=
ERROR: Error while resuming backups, aborting
The attribute 12 requested but not present error is symptomatic of synology BTRFS, and happens with btrfs receive, but is not really an issue. Any way to prevent it being flagged as an issue, and to continue the backups? I suppose no, but who knows ;)
Hi, we are having the same problem with attribute 12 requested but not present. The snapshots seem to be fine otherwise, but btrbk recognizes this as an error and e.g. doesn't delete the source snapshot because of that.
ERROR: ... attribute 12 requested but not present.
ERROR: Error while resuming backups, aborting
It would be great if this could be changed to a warning instead of an error.
@foux It seems to me that synology does not support ACL's in btrfs, but your send-stream contains ACLs in /var/log/journal. Either the target was mounted with noacl option, or the kernel option CONFIG_BTRFS_FS_POSIX_ACL is not enabled on the receiving side. Synology probably does not have it enabled.
attribute 12 refers to BTRFS_SEND_A_OTIME in the send stream (send.h line 121), which I believe is the file "birth" time (see article on LWN). Not sure where that might come from, or why it is expected to be in the stream. Synology is known to use patched versions, I fear they must have done something in that regard in order to support windows shares / BSD / whatever.
It would be great if this could be changed to a warning instead of an error.
if btrfs receive returns an error, this can't be ignored unless it's absolutely clear that nothing breaks by doing so. That said, here's a patch you can try:
diff --git a/btrbk b/btrbk
index b038fff..1104757 100755
--- a/btrbk
+++ b/btrbk
@@ -1580,7 +1580,7 @@ sub btrfs_send_receive($$;$$$)
cmd => vinfo_cmd($target, "btrfs receive", @receive_options, { unsafe => $target_path . '/' } ),
rsh => vinfo_rsh($target, disable_compression => $stream_options->{stream_compress}),
name => "btrfs receive",
- fatal_stderr => sub { m/^ERROR: /; }, # NOTE: btrfs-progs < 4.11: if "btrfs send" fails, "btrfs receive" returns 0!
+ fatal_stderr => sub { WARN "Synology hack, ignoring ERROR: $1" if(s/^ERROR: (attribute 12 requested but not present.*)//); m/^ERROR: /; },
};
my $send_receive_error = 0;
@digint I had the exact same problem as described in this issue (also backing up to a Synology NAS).
I applied your patch (didn't apply cleanly anymore after half a year, the line numbers changed a bit) and can confirm that it workarounds the Synology bug perfectly fine. Thanks a lot!
Therefore, feel free to add
Tested-by: Björn Daase <bjoern@daase.net>
I am looking forward to having a solution like that in master :)
Added a configuration option compat ignore_receive_errors on ignore-receive-errors branch.
Setting this option will ignore all errors from btrfs receive, by setting --max-errors=0. Warnings are printed instead.
included in btrbk-0.32.2