[fix] weed fix built an index with the expected size smaller than the actual size
kmlebedev opened this issue · comments
Describe the bug
ls
-rw-r--r-- 1 seaweed seaweed 16M Apr 18 13:33 /data/bucket_231.dat
volume.list
volume id:231 size:16047320 collection:\"bucket\" file_count:4 delete_count:2 deleted_byte_count:16046990 replica_placement:100 version:3 ttl:261 compact_revision:149 modified_at_second:1713447196
log:
E0418 15:10:28.454905 volume_grpc_vacuum.go:65 failed compact volume 231: volume 231 unexpected new data size: 88 does not match size of content minus deleted: 128
I0418 15:10:28.455663 volume_vacuum.go:174 Cleaning up volume 231 vacuuming...
weed fix built an index with the expected size smaller than the actual size, because of this volume switches to read-only mode
after weed fix
volume.list
volume id:231 size:16047320 collection:\"bucket\" file_count:2 read_only:true replica_placement:100 version:3 ttl:261 compact_revision:149 modified_at_second:1713447196
log
I0418 17:23:53.324309 volume_loading.go:91 readSuperBlock volume 231 version 3
W0418 17:23:53.324343 volume_checking.go:121 data file /data/bucket_231.dat actual 16047320 bytes expected 200 bytes!
I0418 17:23:53.324351 volume_loading.go:128 volumeDataIntegrityChecking failed data file /data/bucket_231.dat actual 16047320 bytesexpected 200 bytes
I0418 17:23:53.324366 needle_map_sorted_file.go:24 Start to Generate /data/bucket_231.sdx from /data/bucket_231.idx
I0418 17:23:53.324783 needle_map_sorted_file.go:26 Finished Generating /data/bucket_231.sdx from /data/bucket_231.idx
I0418 17:23:53.324819 disk_location.go:182 data file /data/bucket_231.dat, replication=100 v=3 size=16047320 ttl=1M
I0418 17:23:53.324826 store.go:471 mount volume 231
see idx:
./see_idx -volumeId 231
key:fc138c121e6e6b6c offset:1 size:82(82 B)
key:fc138c121e7c6fef offset:15 size:46(46 B)
see dat
./see_dat -volumeId 231
I0419 00:11:56.464368 volume_loading.go:91 readSuperBlock volume 231 version 3
I0419 00:11:56.467121 see_dat.go:36 231,fc138c121e6e6b6cad4c01b3 offset 8 size 82(82 B) cookie ad4c01b3 appendedAt 2024-03-19 14:38:00.05674303 +0500 +05 name secuirty-ansible-galaxy-renovate-non_protected
I0419 00:11:56.467135 see_dat.go:36 231,fc138c121e7c6fefefb29642 offset 120 size 46(46 B) cookie efb29642 appendedAt 2024-03-22 13:58:41.595410049 +0500 +05 name -protected
I0419 00:11:56.474538 see_dat.go:36 231,fc138c121ecd24bdca1a808d offset 200 size 8023495(7.65 MiB) cookie ca1a808d appendedAt 2024-04-17 20:22:04.112901039 +0500 +05 name projects-non_protected
I0419 00:11:56.480959 see_dat.go:36 231,fc138c121ed0f05ca5ff77a6 offset 8023728 size 8023495(7.65 MiB) cookie a5ff77a6 appendedAt 2024-04-18 18:32:45.305981473 +0500 +05 name projects-non_protected
I0419 00:11:56.480976 see_dat.go:36 231,fc138c121ecd24bd00000000 offset 16047256 size 0(0 B) cookie 00000000 appendedAt 2024-04-18 18:32:46.253828824 +0500 +05 name
I0419 00:11:56.480981 see_dat.go:36 231,fc138c121ed0f05c00000000 offset 16047288 size 0(0 B) cookie 00000000 appendedAt 2024-04-18 18:33:16.590179944 +0500 +05 name
hexdump -n 200 -C /tmp/231.dat
00000000 03 64 01 05 00 95 00 00 ad 4c 01 b3 fc 13 8c 12 |.d.......L......|
00000010 1e 6e 6b 6c 00 00 00 52 00 00 00 16 50 4b 05 06 |.nkl...R....PK..|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 1e 2e 73 65 63 75 69 72 74 79 2d 61 6e 73 |....secuirty-ans|
00000040 69 62 6c 65 2d 67 61 6c 61 78 79 2d 72 65 6e 6f |ible-galaxy-reno|
00000050 76 61 74 65 2d 6e 6f 6e 5f 70 72 6f 74 65 63 74 |vate-non_protect|
00000060 65 64 00 00 65 f9 5c f8 01 05 e2 4f 23 93 17 be |ed..e.\....O#...|
00000070 20 e5 db ed 84 76 00 00 ef b2 96 42 fc 13 8c 12 | ....v.....B....|
00000080 1e 7c 6f ef 00 00 00 2e 00 00 00 16 50 4b 05 06 |.|o.........PK..|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 00 00 1e 0a 2d 70 72 6f 74 65 63 74 65 64 00 00 |....-protected..|
000000b0 65 fd 48 41 01 05 e2 4f 23 93 17 bf 0a 7e 70 9a |e.HA...O#....~p.|
000000c0 86 81 00 00 00 2e 00 00 |........|
000000c8
ls cpd
-rw-r--r-- 1 seaweed seaweed 88 Apr 18 18:53 /data/bucket/231.cpd
see cpd
I0419 00:22:54.506440 volume_loading.go:91 readSuperBlock volume 231 version 3
I0419 00:22:54.508873 see_dat.go:36 231,fc138c121e7c6fefefb29642 offset 8 size 46(46 B) cookie efb29642 appendedAt 2024-03-22 13:58:41.595410049 +0500 +05 name -protected
hexdump -C /tmp/231.cpd
00000000 03 64 01 05 00 96 00 00 ef b2 96 42 fc 13 8c 12 |.d.........B....|
00000010 1e 7c 6f ef 00 00 00 2e 00 00 00 16 50 4b 05 06 |.|o.........PK..|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 1e 0a 2d 70 72 6f 74 65 63 74 65 64 00 00 |....-protected..|
00000040 65 fd 48 41 01 05 e2 4f 23 93 17 bf 0a 7e 70 9a |e.HA...O#....~p.|
00000050 86 81 00 00 00 2e 00 00 |........|
00000058
see cpx:
key:fc138c121e7c6fef offset:1 size:46(46 B)
System Setup
weed version
version 8000GB 3.65 0edc5aad9a489d0c99213a45e27654703fa5da70 linux amd64
Expected behavior
weed fix it is necessary to create an index at which compact will not cause errors.
I remember the problem when the deleted file is counted twice, so we see the file_count field as 4, although in fact there are 2.
It turns out that there are no current files in the dat file.
Also tried to do compact manually
For method=0, it compacts based on the .dat file, works if .idx file is corrupted.
For method=1, it compacts based on the .idx file, works if deletion happened but not written to .dat files.
weed compact -collection bucket -volumeId 231 -dir=/data
I0418 18:53:14.938912 volume_loading.go:91 readSuperBlock volume 231 version 3
W0418 18:53:14.939216 volume_checking.go:121 data file bucket_231.dat actual 16047320 bytes expected 200 bytes!
I0418 18:53:14.939332 volume_loading.go:128 volumeDataIntegrityChecking failed data file bucket_231.dat actual 16047320 bytes expected 200 bytes
I0418 18:53:14.939837 volume_loading.go:91 readSuperBlock volume 231 version 3
weed compact -collection bucket -volumeId 231 -method=1 -dir=/data
I0418 18:53:56.382445 volume_loading.go:91 readSuperBlock volume 231 version 3
W0418 18:53:56.382699 volume_checking.go:121 data file bucket_231.dat actual 16047320 bytes expected 200 bytes!
I0418 18:53:56.382835 volume_loading.go:128 volumeDataIntegrityChecking failed data file bucket_231.dat actual 16047320 bytes expected 200 bytes
F0418 18:53:56.385220 compact.go:54 Compact Volume [ERROR] volume 231 unexpected new data size: 88 does not match size of content minus deleted: 128
goroutine 1 [running]:
github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0)
/github/workspace/weed/glog/glog.go:768 +0x85
github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x4027380, 0x3, 0xc0005edd50, {0x32de15a?, 0xc000da9d28?}, 0x1?, 0x0)
/github/workspace/weed/glog/glog.go:719 +0x38a
github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printf(0x4027380, 0x3, {0x24aa0bb, 0x1a}, {0xc000da9d28, 0x1, 0x1})
/github/workspace/weed/glog/glog.go:657 +0x10a
github.com/seaweedfs/seaweedfs/weed/glog.Fatalf(...)
/github/workspace/weed/glog/glog.go:1154
github.com/seaweedfs/seaweedfs/weed/command.runCompact(0x4008738?, {0xc0000500a0?, 0x6?, 0x6?})
/github/workspace/weed/command/compact.go:54 +0x209
main.main()
/github/workspace/weed/weed.go:95 +0x3d9
@chrislusf It looks like, judging by the index, after weed fix
there should be two files left, but in fact after weed compact
there remains one file(-protected) and the second(secuirty-ansible-galaxy-renovate-non_protected) one was unexpectedly deleted:
I0419 00:11:56.467121 see_dat.go:36 231,fc138c121e6e6b6cad4c01b3 offset 8 size 82(82 B) cookie ad4c01b3 appendedAt 2024-03-19 14:38:00.05674303 +0500 +05 name secuirty-ansible-galaxy-renovate-non_protected
display additional TTL values is 1M
./see_dat -volumeId 231
I0427 18:13:32.053612 volume_loading.go:91 readSuperBlock volume 231 version 3
I0427 18:13:32.055783 see_dat.go:36 231,fc138c121e6e6b6cad4c01b3 offset 8 size 82(82 B) cookie ad4c01b3 appendedAt 2024-03-19 14:38:00.057043092 +0500 +05 name secuirty-ansible-galaxy-renovate-non_protected flags 0000001e ttl 1M
I0427 18:13:32.055811 see_dat.go:36 231,fc138c121e7c6fefefb29642 offset 120 size 46(46 B) cookie efb29642 appendedAt 2024-03-22 13:58:41.595009609 +0500 +05 name -protected flags 0000001e ttl 1M
accordingly, the compactor deletes files in accordance with TTL, and the indexer does not take into account TTL
Compaction would not run on TTL volumes. We should add some logic to disable it by default in weed compact
also.