[BUG] Healthy volume suddenly comes up formatted

[wirr00]

Describe the bug

A healthy volume (RWX, created by PVC with a retain policy) using ext4 file system without generating any errors suddenly comes up empty.
dumpe2fs shows that the file system was newly created:

dumpe2fs /dev/longhorn/media-volume | head -n 40
dumpe2fs 1.46.2 (28-Feb-2021)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          e45c76d1-3069-4c95-a4ef-4c13751bdd00
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         unsigned_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              78643200
Block count:              314572800
Reserved block count:     0
Overhead clusters:        5218869
Free blocks:              309353924
Free inodes:              78643189
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Tue Jun 11 09:05:05 2024
Last mount time:          Fri Jun 14 18:46:51 2024
Last write time:          Fri Jun 14 18:47:05 2024
Mount count:              3
Maximum mount count:      -1
Last checked:             Fri Jun 14 13:03:31 2024
Check interval:           0 (<none>)
Lifetime writes:          42 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     32

To Reproduce

• Create ext4-volume of 1.2TiB, 2 replicas, RWX by PVC
• Attach volume to several pods
• Wait (possibly > 10 months)

Expected behavior

Volume doesn't get formatted without explicit user interaction.

Support bundle for troubleshooting

supportbundle_54ffb2a5-9c16-4485-9f07-4c0b8f293f73_2024-06-15T14-53-45Z.zip

Environment

• Longhorn version: 1.61
• Impacted volume (PV): media-pv
• Installation method: kubectl
• Kubernetes distro: kubeadm 1.28.2
  • Number of control plane nodes in the cluster: 3
  • Number of worker nodes in the cluster: 3
• Node config
  • OS type and version: Armbian 23.8.1 bullseye
  • Kernel version: 5.10.110-rockchip-rk3588
  • CPU per node: 8
  • Memory per node: 16GiB
  • Disk type: m2 SSD
  • Network bandwidth between the nodes (Gbps): 1gbps
• Underlying Infrastructure: Bare Metal
• Number of Longhorn volumes in the cluster: 8