Scrub never completes after rsyncing from .zfs/snapshot dir
brenc opened this issue · comments
System information
| Type | Version/Name |
|---|---|
| Distribution Name | FreeBSD |
| Distribution Version | 14.1-RELEASE-p1 |
| Kernel Version | 14.1-RELEASE |
| Architecture | x86_64 |
| OpenZFS Version | zfs-2.2.4-FreeBSD_g256659204, zfs-kmod-2.2.4-FreeBSD_g256659204 |
Describe the problem you're observing
We have a remote backup process that uses rsync to back up files from a server using snapshots in the .zfs/snapshot directories. I noticed that after I started using this process subsequent scrubs (which run once per month on the 1st) never complete. I have been troubleshooting this for months and can confidently say that accessing files from these snapshot dirs is what triggers this bug. If I stop the remote backup process, reboot the server, and then run a scrub it completes normally in < 12 hours. As soon as I start the remote backup backup process, subsequent scrubs that run never complete even if left running for 5+ days.
I have tried pausing and resuming the scrub as well as stopping the scrub completely and restarting it. Nothing seems to correct this issue except a reboot. It always gets to around 70-80ish% done and hangs unless I reboot.
Some additional observations: scrub speed appears to be much reduced. This is what it looks like while stuck:
pool: storage
state: ONLINE
scan: scrub in progress since Mon Jul 1 00:00:00 2024
73.3T / 102T scanned at 595M/s, 73.3T / 102T issued at 595M/s
0B repaired, 71.81% done, 14:05:39 to go
While a normal scrub will run at ~3 G/s. There is definitely still high read IO on the drives:
capacity operations bandwidth syncq_read syncq_write asyncq_read asyncq_write scrubq_read trimq_write rebuildq_write
pool alloc free read write read write pend activ pend activ pend activ pend activ pend activ pend activ pend activ
-------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
storage 102T 291T 4.87K 0 52.9M 0 0 0 0 0 0 0 97 48 0 0 0 0 0 0
raidz2-0 25.6T 72.6T 1.19K 0 12.6M 0 0 0 0 0 0 0 18 12 0 0 0 0 0 0
DRIVE - - 192 0 1.88M 0 0 0 0 0 0 0 5 2 0 0 0 0 0 0
DRIVE - - 213 0 2.30M 807 0 0 0 0 0 0 2 2 0 0 0 0 0 0
DRIVE - - 180 0 1.72M 807 0 0 0 0 0 0 1 2 0 0 0 0 0 0
DRIVE - - 203 0 1.99M 807 0 0 0 0 0 0 1 2 0 0 0 0 0 0
DRIVE - - 231 0 2.48M 807 0 0 0 0 0 0 1 2 0 0 0 0 0 0
DRIVE - - 194 0 2.21M 807 0 0 0 0 0 0 2 2 0 0 0 0 0 0
raidz2-1 25.6T 72.6T 1.27K 0 13.9M 3.94K 0 0 0 0 0 0 49 12 0 0 0 0 0 0
DRIVE - - 202 0 2.34M 807 0 0 0 0 0 0 9 2 0 0 0 0 0 0
DRIVE - - 214 0 2.26M 807 0 0 0 0 0 0 6 2 0 0 0 0 0 0
DRIVE - - 202 0 1.94M 807 0 0 0 0 0 0 5 2 0 0 0 0 0 0
DRIVE - - 198 0 1.88M 1.58K 0 0 0 0 0 0 2 2 0 0 0 0 0 0
DRIVE - - 253 0 2.61M 1.58K 0 0 0 0 0 0 1 2 0 0 0 0 0 0
DRIVE - - 226 0 2.87M 1.58K 0 0 0 0 0 0 9 2 0 0 0 0 0 0
raidz2-2 25.6T 72.6T 1.10K 2 12.1M 9.47K 0 0 0 0 0 0 72 12 0 0 0 0 0 0
DRIVE - - 211 0 2.18M 1.58K 0 0 0 0 0 0 12 2 0 0 0 0 0 0
DRIVE - - 172 0 2.06M 1.58K 0 0 0 0 0 0 12 2 0 0 0 0 0 0
DRIVE - - 189 0 2.17M 5.52K 0 0 0 0 0 0 10 2 0 0 0 0 0 0
DRIVE - - 207 0 2.13M 1.58K 0 0 0 0 0 0 13 2 0 0 0 0 0 0
DRIVE - - 161 0 1.76M 1.58K 0 0 0 0 0 0 13 2 0 0 0 0 0 0
DRIVE - - 187 0 1.84M 1.58K 0 0 0 0 0 0 9 2 0 0 0 0 0 0
raidz2-3 25.6T 72.6T 1.31K 4 14.3M 31.6K 0 0 0 0 0 0 15 12 0 0 0 0 0 0
DRIVE - - 205 0 2.01M 3.94K 0 0 0 0 0 0 2 2 0 0 0 0 0 0
DRIVE - - 215 0 2.28M 3.94K 0 0 0 0 0 0 3 2 0 0 0 0 0 0
DRIVE - - 235 0 2.46M 6.31K 0 0 0 0 0 0 5 2 0 0 0 0 0 0
DRIVE - - 218 0 2.37M 9.47K 0 0 0 0 0 0 3 2 0 0 0 0 0 0
DRIVE - - 220 0 2.49M 3.94K 0 0 0 0 0 0 1 2 0 0 0 0 0 0
DRIVE - - 242 0 2.67M 3.94K 0 0 0 0 0 0 1 2 0 0 0 0 0 0
-------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
The scrub appears to be running even when no progress is made. % done never advances.
ZFS replication (this server pulls from the primary) essentially grinds to a halt as well.
Pool config:
# zpool get all storage
NAME PROPERTY VALUE SOURCE
storage size 393T -
storage capacity 26% -
storage altroot - default
storage health ONLINE -
storage guid 15571001037900998829 -
storage version - default
storage bootfs - default
storage delegation on default
storage autoreplace off default
storage cachefile - default
storage failmode wait default
storage listsnapshots off default
storage autoexpand off default
storage dedupratio 1.00x -
storage free 291T -
storage allocated 102T -
storage readonly off -
storage ashift 12 local
storage comment - default
storage expandsize - -
storage freeing 0 -
storage fragmentation 3% -
storage leaked 0 -
storage multihost off default
storage checkpoint - -
storage load_guid 4612684389734077050 -
storage autotrim off default
storage compatibility off default
storage bcloneused 0 -
storage bclonesaved 0 -
storage bcloneratio 1.00x -
storage feature@async_destroy enabled local
storage feature@empty_bpobj active local
storage feature@lz4_compress active local
storage feature@multi_vdev_crash_dump enabled local
storage feature@spacemap_histogram active local
storage feature@enabled_txg active local
storage feature@hole_birth active local
storage feature@extensible_dataset active local
storage feature@embedded_data active local
storage feature@bookmarks enabled local
storage feature@filesystem_limits enabled local
storage feature@large_blocks active local
storage feature@large_dnode enabled local
storage feature@sha512 enabled local
storage feature@skein enabled local
storage feature@edonr enabled local
storage feature@userobj_accounting active local
storage feature@encryption active local
storage feature@project_quota active local
storage feature@device_removal enabled local
storage feature@obsolete_counts enabled local
storage feature@zpool_checkpoint enabled local
storage feature@spacemap_v2 active local
storage feature@allocation_classes enabled local
storage feature@resilver_defer enabled local
storage feature@bookmark_v2 enabled local
storage feature@redaction_bookmarks enabled local
storage feature@redacted_datasets enabled local
storage feature@bookmark_written enabled local
storage feature@log_spacemap active local
storage feature@livelist enabled local
storage feature@device_rebuild enabled local
storage feature@zstd_compress enabled local
storage feature@draid enabled local
storage feature@zilsaxattr active local
storage feature@head_errlog active local
storage feature@blake3 enabled local
storage feature@block_cloning enabled local
storage feature@vdev_zaps_v2 active local
I don't know if it matters but I'm using encryption. Here are the encryption root settings:
# zfs get all storage/e
NAME PROPERTY VALUE SOURCE
storage/e type filesystem -
storage/e creation Wed Sep 6 15:10 2023 -
storage/e used 68.2T -
storage/e available 193T -
storage/e referenced 432K -
storage/e compressratio 1.05x -
storage/e mounted yes -
storage/e quota none default
storage/e reservation none default
storage/e recordsize 1M inherited from storage
storage/e mountpoint /storage/e default
storage/e sharenfs off default
storage/e checksum on default
storage/e compression lz4 inherited from storage
storage/e atime off inherited from storage
storage/e devices on default
storage/e exec off inherited from storage
storage/e setuid off inherited from storage
storage/e readonly off default
storage/e jailed off default
storage/e snapdir visible inherited from storage
storage/e aclmode discard default
storage/e aclinherit restricted default
storage/e createtxg 62704 -
storage/e canmount on default
storage/e xattr on default
storage/e copies 1 default
storage/e version 5 -
storage/e utf8only off -
storage/e normalization none -
storage/e casesensitivity sensitive -
storage/e vscan off default
storage/e nbmand off default
storage/e sharesmb off default
storage/e refquota none default
storage/e refreservation none default
storage/e guid 14413502527107633131 -
storage/e primarycache all default
storage/e secondarycache all default
storage/e usedbysnapshots 0B -
storage/e usedbydataset 432K -
storage/e usedbychildren 68.2T -
storage/e usedbyrefreservation 0B -
storage/e logbias latency default
storage/e objsetid 282 -
storage/e dedup off default
storage/e mlslabel none default
storage/e sync standard default
storage/e dnodesize legacy default
storage/e refcompressratio 1.00x -
storage/e written 432K -
storage/e logicalused 70.7T -
storage/e logicalreferenced 103K -
storage/e volmode default default
storage/e filesystem_limit none default
storage/e snapshot_limit none default
storage/e filesystem_count none default
storage/e snapshot_count none default
storage/e snapdev hidden default
storage/e acltype nfsv4 default
storage/e context none default
storage/e fscontext none default
storage/e defcontext none default
storage/e rootcontext none default
storage/e relatime on default
storage/e redundant_metadata all default
storage/e overlay on default
storage/e encryption aes-256-gcm -
storage/e keylocation file:///etc/zfs_encryption_key local
storage/e keyformat passphrase -
storage/e pbkdf2iters 350000 -
storage/e encryptionroot storage/e -
storage/e keystatus available -
storage/e special_small_blocks 0 default
storage/e prefetch all default
Describe how to reproduce the problem
rsync files from .zfs/snapshot directories then run a scrub.
Include any warning/errors/backtraces from the system logs
There are no relevant logs as far as I can tell.
I believe that's all the information I have available on this right now. Please let me know if you need anything else.