Failed Lease. Can't leave or join VG
TProhofsky opened this issue · comments
Tom Prohofsky commented
One of my server nodes in the cluster crashed while part of a VG. Some time went by and one of the other nodes lost their lease. Now I have an active LV on a node but no lock for the VG of that LV:
[root@node3 ~]# lvs
VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
Reading VG sbvg_drbdpool without a lock.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
csilv2eg47tjvu91qf sbvg_datalake rwi---r--- 100.00g
csilvnglz2r7n6i85 sbvg_datalake rwi---r--- 1000.00g
csilvpvr6bs1lps8u sbvg_datalake -wi------- 100.00g
poctest_00000 sbvg_drbdpool Vwi-a-tz-- 50.01g thinpool 5.74
thinpool sbvg_drbdpool twi-aotz-- 931.31g 0.31 10.55
[root@node3 ~]#
Attemping to start the lock completes immediately but has no effect:
[root@node3 ~]# /usr/sbin/vgchange --lock-start --lock-opt auto sbvg_drbdpool
Starting locking. Waiting until locks are ready...
[root@node3 ~]# lvs
VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
Reading VG sbvg_drbdpool without a lock.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
csilv2eg47tjvu91qf sbvg_datalake rwi---r--- 100.00g
csilvnglz2r7n6i85 sbvg_datalake rwi---r--- 1000.00g
csilvpvr6bs1lps8u sbvg_datalake -wi------- 100.00g
poctest_00000 sbvg_drbdpool Vwi-a-tz-- 50.01g thinpool 5.74
thinpool sbvg_drbdpool twi-aotz-- 931.31g 0.31 10.55
[root@node3 ~]#
Attempting to stop the locks thinking that might clear it shows this:
[root@node3 ~]# /usr/sbin/vgchange --lock-stop sbvg_drbdpool
VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
Reading VG sbvg_drbdpool without a lock.
VG sbvg_drbdpool stop failed: LVs must first be deactivated
Drop and force did not clear issue:
lvmlockctl --drop sbvg_datalake
vgchange --lock-type none --lockopt force sbvg_drbdpool
vgchange --lock-start --lock-opt auto sbvg_drbdpool
Tom Prohofsky commented
Attempting to try it again after Node 1's issue with systemd was fix showed this:
[root@node3 ~]# /usr/sbin/vgchange --lock-start --lock-opt auto sbvg_drbdpool
Starting locking. Waiting until locks are ready...
[root@node3 ~]# lvs
Broadcast message from systemd-journald@node3 (Thu 2020-09-17 07:47:33 CDT):
lvmlockctl[15154]: Lost access to sanlock lease storage in VG sbvg_datalake.
Broadcast message from systemd-journald@node3 (Thu 2020-09-17 07:47:33 CDT):
lvmlockctl[15154]: Immediately deactivate LVs in VG sbvg_datalake.
Broadcast message from systemd-journald@node3 (Thu 2020-09-17 07:47:33 CDT):
lvmlockctl[15154]: Once VG is unused, run lvmlockctl --drop sbvg_datalake.
Message from syslogd@node3 at Sep 17 07:47:33 ...
lvmlockctl:Immediately deactivate LVs in VG sbvg_datalake.
Message from syslogd@node3 at Sep 17 07:47:33 ...
lvmlockctl:Once VG is unused, run lvmlockctl --drop sbvg_datalake.
VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
Reading VG sbvg_drbdpool without a lock.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
csilv2eg47tjvu91qf sbvg_datalake rwi---r--- 100.00g
csilvnglz2r7n6i85 sbvg_datalake rwi---r--- 1000.00g
csilvpvr6bs1lps8u sbvg_datalake -wi------- 100.00g
poctest_00000 sbvg_drbdpool Vwi-a-tz-- 50.01g thinpool 5.74
thinpool sbvg_drbdpool twi-aotz-- 931.31g 0.31 10.55
Tom Prohofsky commented