Giters

seaweedfs / seaweedfs

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Metacache: find entry error: get /xxx : leveldb: closed

eliphatfs opened this issue · comments

commented

Describe the bug
During user usage of FUSE mounts, the following error is seen:

Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: E0316 03:56:31.624770 meta_cache.go:76 Metacache: find entry error: get /ariesdv0 : leveldb: closed
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: F0316 03:56:31.624804 filer_pb_tail.go:91 process directory:"/"  event_notification:{old_entry:{name:"ariesdv0"  is_directory:>
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: goroutine 262 [running]:
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0)
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/glog/glog.go:768 +0x85
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x724eb60, 0x3, 0xc018294070, {0x590e3b9?, 0xc000dd9998?}, 0x2?, 0>
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/glog/glog.go:719 +0x38a
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printf(0x724eb60, 0x3, {0x4219de6, 0xe}, {0xc000dd9998, 0x2, 0x2})
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/glog/glog.go:657 +0x10a
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/glog.Fatalf({0x4219de6?, 0x4ce6078?}, {0xc000dd9998?, 0xc0008c4aa0?, 0x0?})
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/glog/glog.go:1154 +0x38
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata.makeSubscribeMetadataFunc.func1({0x4d22c78, 0xc000c2f320})
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/pb/filer_pb_tail.go:91 +0x39a
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1.1(0xc001143008)
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/mount/wfs_filer_client.go:27 +0x68
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/pb.WithGrpcClient(0xbd?, 0x46dfa25b?, 0xc000e4dce8, {0xc0002ec6c0, 0x11}, 0x0?, {0xc001148>
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/pb/grpc_client_server.go:155 +0x343
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1()
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/mount/wfs_filer_client.go:25 +0x155
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/util.Retry({0x420aa72, 0xa}, 0xc007a05e00)
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/util/retry.go:16 +0xa7
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient(0xc00037da00?, 0x78?, 0x4115be?)
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/mount/wfs_filer_client.go:18 +0x65
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata({0x4cd2f48, 0xc000622000}, 0xc00037da00, 0xc000a1c120)
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/pb/filer_pb_tail.go:50 +0x87
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2()
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:74 +0x26
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/util.RetryUntil({0x4227d36, 0x11}, 0xc007a05f58, 0x444ab50)
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/util/retry.go:64 +0x4e
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents(0xc00101d260, 0x46dfa25b, {0x4cd2f48, 0xc000622000}, >
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:72 +0x176
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]: created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
Mar 16 03:56:31 aries-00 seaweedfs-client[2202451]:         /github/workspace/weed/mount/weedfs.go:123 +0x11d
Mar 16 03:56:31 aries-00 systemd[1]: seaweedclient.service: Main process exited, code=exited, status=255/EXCEPTION

System Setup

  • /usr/local/bin/weed server -volume=0 -filer -dir=/weedfs
  • /usr/local/bin/weed volume -max=400 -dir=/weedfs -mserver=10.8.149.13:9333 on 8 machines different from the master
  • These commands are done by systemctl services with setups exactly as the wiki page.
  • OS version: Ubuntu 22.04
  • output of weed version: version 8000GB 3.63 54d7748 linux amd64
  • if using filer, show the content of filer.toml: no filer.toml
  • security:
[access]
ui = false

[grpc]
ca = "/etc/ariesdockerd/certs/Aries_SeaweedFS_CA.crt"

[grpc.volume]
cert = "/etc/ariesdockerd/certs/volume01.crt"
key  = "/etc/ariesdockerd/certs/volume01.key"

[grpc.master]
cert = "/etc/ariesdockerd/certs/master01.crt"
key  = "/etc/ariesdockerd/certs/master01.key"

[grpc.filer]
cert = "/etc/ariesdockerd/certs/filer01.crt"
key  = "/etc/ariesdockerd/certs/filer01.key"

[grpc.client]
cert = "/etc/ariesdockerd/certs/client01.crt"
key  = "/etc/ariesdockerd/certs/client01.key"

Expected behavior
It will not crash.