[SNSD] Unexpected EOF when downloading File
technofetishism opened this issue · comments
NOTE
We have a single part file stored on minio. When attempting to download it, use mc cat, or other methods of reading the file via minio, the stream terminates with an
mcli: <ERROR> Unable to read from `minio/gitlab-lfs/af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee`. unexpected EOF.
Inspecting the file on the underlying filesystem shows that the file is not truncated in comaprision to the files metadata
mcli stat minio/gitlab-lfs/af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee Name : 88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee Date : 2024-04-24 10:05:21 UTC Size : 2.3 MiB ETag : cb28bbfea8a830ad0c0a879f5c6c8ead Type : file Metadata : Content-Type: binary/octet-stream
/data/minio/gitlab-lfs/af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee/6bb0aeef-6cb4-4f36-9147-d1750e517904$ du -h part.1 2.4M part.1
and appears to be a difference expected by the added information by minio.
Investigating the file in the backend where the downloads terminate shows the following (this is in vim)
DH002940 308820.000 7370360.000 SURV 1009.855 -25.099 -12.205 1^\ßYf7Ç%´®¾wá<99>·ª&@<98>!?ÂPÖqÖ<8f>µ$¾Ôà'034.953 ^M
The original file is
DH002940 308820.000 7370360.000 SURV 1009.855 -25.099 -12.205 1034.953 ^M
and the download fails exactly at
DH002940 308820.000 7370360.000 SURV 1009.855 -25.099 -12.205 1mcli: <ERROR> Unable to read fromminio/gitlab-lfs/af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee`. unexpected EOF.
Right where the control code shows up `^'
Running mcli admin trace -a -v gives the following with the opinion of minio the
127.0.0.1:9000 [OS os.OpenFileR] [2024-05-02T05:23:37.928] /data/minio/gitlab-lfs/af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee/xl.meta 53.928µs
127.0.0.1:9000 [STORAGE storage.ReadXL] [2024-05-02T05:23:37.928] /data/minio gitlab-lfs af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee total-errs-availability=0 total-errs-timeout=0 106.591µs
127.0.0.1:9000 [OS os.OpenFileR] [2024-05-02T05:23:37.930] /data/minio/gitlab-lfs/af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee/6bb0aeef-6cb4-4f36-9147-d1750e517904/part.1 30.161µs
127.0.0.1:9000 [STORAGE storage.ReadFileStream] [2024-05-02T05:23:37.930] /data/minio gitlab-lfs af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee/6bb0aeef-6cb4-4f36-9147-d1750e517904/part.1 total-errs-availability=0 total-errs-timeout=0 96.237µs
s3.company.io:9000 [REQUEST s3.GetObject] [2024-05-02T05:23:37.928] [Client IP: 10.15.130.3]
s3.company.io:9000 GET /gitlab-lfs/af/6e/88db16ce4eb70e9445dc2aa499d352404cf03447a88ab92900ca8889d9ee?response-content-disposition=attachment%3B%20filename%3D%22qerlcfly2z.map%22%3B%20filename%2A%3DUTF-8%27%27qerlcfly2z.map&response-content-type=application%2Foctet-stream&X-Amz-Expires=600&X-Amz-Date=20240502T052337Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=F20240502%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=0c033d6586a28e159a72fd305325c60fa800ea8b10fb7cedfd3cc627c8e257d2
s3.company.io:9000 Proto: HTTP/1.1
s3.company.io:9000 Host: s3.company.io:9000
s3.company.io:9000 Accept-Encoding: gzip, deflate, br
s3.company.io:9000 Connection: keep-alive
s3.company.io:9000 Sec-Fetch-Mode: navigate
s3.company.io:9000 Sec-Fetch-Site: same-site
s3.company.io:9000 Sec-Fetch-User: ?1
s3.company.io:9000 X-Amz-Signature-Age: 929
s3.company.io:9000 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,/;q=0.8
s3.company.io:9000 Content-Length: 0
s3.company.io:9000 Upgrade-Insecure-Requests: 1
s3.company.io:9000 Sec-Fetch-Dest: document
s3.company.io:9000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0
s3.company.io:9000 Accept-Language: en-US,en;q=0.5
s3.company.io:9000 Cookie: token=
s3.company.io:9000 Referer: https://gitlab.company.io/
s3.company.io:9000
s3.company.io:9000 [RESPONSE] [2024-05-02T05:23:38.078] [ Duration 150.492ms TTFB 2.925007ms ↑ 202 B ↓ 1.0 MiB ]
s3.company.io:9000 200 OK
s3.company.io:9000 Server: MinIO
s3.company.io:9000 Vary: Origin,Accept-Encoding
s3.company.io:9000 Content-Disposition: attachment; filename="qerlcfly2z.map"; filename*=UTF-8''qerlcfly2z.map
s3.company.io:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
s3.company.io:9000 X-Xss-Protection: 1; mode=block
s3.company.io:9000 Accept-Ranges: bytes
s3.company.io:9000 Content-Length: 2434097
s3.company.io:9000 Content-Type: application/octet-stream
s3.company.io:9000 Last-Modified: Wed, 24 Apr 2024 10:05:21 GMT
s3.company.io:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
s3.company.io:9000 X-Amz-Request-Id: 17CB948D57BF77F7
s3.company.io:9000 ETag: "cb28bbfea8a830ad0c0a879f5c6c8ead"
s3.company.io:9000 X-Content-Type-Options: nosniff
s3.company.io:9000
s3.company.io:9000
Your Environment
- Version used (
minio --version): RELEASE.2024-05-01T01-11-10Z - Server setup and configuration: Single disk single node VM running as systemd service
- Operating System and version (
uname -a):6.5.0-26-generic #26~22.04.1-Ubuntu
Is your backend filesystem ext4?
backend filesystem is on xfs
Part files have streaming checksums. If your download terminates with unexpected EOF at a boundary, a chunk failed the checksum. If you have no redundancy there is no way to reconstruct this.
backend filesystem is on xfs
It looks like your data is bit-rotted, as @klauspost explained.
It would help if you had redundancy for this to reconstruct; otherwise, using a single drive, you are at the mercy of the drive.