Giters

seaweedfs / seaweedfs

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

`weed filer.copy` and `weed benchmark` sometimes shows `wrong jwt` error.

eliphatfs opened this issue · comments

commented

Describe the bug
I installed a new cluster and benchmarked with weed benchmark. 6 out of 1048576 requests failed with wrong jwt error.
When I run weed filer.copy, there is also this error and seems that filer.copy exited after several of these errors but with code 0 (I am unsure because the output of filer.copy is very noisy with chunking information of large files). Only 7TB out of 40TB files got copied when filer.copy exits.

System Setup

  • /usr/local/bin/weed server -volume=0 -filer -dir=/weedfs
  • /usr/local/bin/weed volume -max=400 -dir=/weedfs -mserver=10.8.149.13:9333 on 8 machines different from the master
  • These commands are done by systemctl services with setups exactly as the wiki page.
  • OS version: Ubuntu 22.04
  • output of weed version: version 30GB 3.62 59b8af9 linux amd64
  • if using filer, show the content of filer.toml: no filer.toml
  • security:
[jwt.signing]
key = "the_same_key"

[jwt.filer_signing]
key = "the_same_key"

[jwt.signing.read]
key = "the_same_key"

[access]
ui = false

[grpc]
ca = "/etc/ariesdockerd/certs/Aries_SeaweedFS_CA.crt"

[grpc.volume]
cert = "/etc/ariesdockerd/certs/volume01.crt"
key  = "/etc/ariesdockerd/certs/volume01.key"

[grpc.master]
cert = "/etc/ariesdockerd/certs/master01.crt"
key  = "/etc/ariesdockerd/certs/master01.key"

[grpc.filer]
cert = "/etc/ariesdockerd/certs/filer01.crt"
key  = "/etc/ariesdockerd/certs/filer01.key"

[grpc.client]
cert = "/etc/ariesdockerd/certs/client01.crt"
key  = "/etc/ariesdockerd/certs/client01.key"

Expected behavior
No error seen. filer.copy should not exit before data gets copied.

Screenshots
image
image

commented

i am seeing maybe related issues with jwt time to expire set to the default 10 sec, when using filer mount, i can write small files without problems, but if i try to write large files (which take longer than 10 sec to write) they fail with JWT error.
If i increase the JWT expiration time to 1 hour, it works without problems. Probably filer should check if it needs to refresh the token for longer writes.