Unable to create a .tar archive bigger than 256MB
fungiboletus opened this issue · comments
Hei,
I have a folder with 300 000 files and Hubic recommends to have 50 000 files maximum. So I want to create a .tar archive directly in the mount point (the files and the archive don't fit on my system).
It doesn't seem to work 😢
Context
- svfs version : 0.9.1
- storage provider : OVH
- product : Hubic
Steps to reproduce this issue :
- Have a folder bigger than 256MB
tar --verbose --create --file /svfs_mount/archive.tar /folder
- Go to the beach and come back when tar has finished. 🏖
Results you expected :
The .tar archive is created and contains all the files.
Results you observed :
Tar exits with return code 2 :
tar: /svfs_mount/archive.tar: Cannot write: Input/output error
tar: Error is not recoverable: exiting now
The archive.tar file exists and is 256MB (268431360).
I've got similar issues (on OVH/hubic).
It seems that the maximum segment size (default mount option segment_size=256
) doesn't work as expected. Any file larger than the maximum segment size is not properly split into multiple segments.
If you mount with option segment_size=10
you will observe the error a lot earlier at an upload of 10Mb.
This currently limits the maximum file size to be about 5Gb (the maximum size of a segment), when mounted with option segment_size=5120
.
As I have a lot of files > 5Gb that I'd like to store on Hubic, this is a blocker for me...
Getting the same issue, also with the suggested rsync method. Basically all files greater that the segmententation limit are broken. Endpoint is hubic.
Can you provide a debug trace ? I suspect this is a duplicate of #100.
How do I generate it?
Use the debug
mount option documented here.
DEBU[2017-09-22T13:58:29+02:00] <- Lookup [ID=0xa2 Node=0x2 Uid=1000 Gid=1000 Pid=22793] "rick.mkv" source=fuse
DEBU[2017-09-22T13:58:29+02:00] -> [ID=0xa2] Lookup error=ENOENT source=fuse
DEBU[2017-09-22T13:58:29+02:00] <- Create [ID=0xa3 Node=0x2 Uid=1000 Gid=1000 Pid=22793] "rick.mkv" fl=OpenWriteOnly+OpenCreate mode=-rw------- umask=--------w- source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa3] Create {0x3 gen=0 valid=1m0s attr={valid=1m0s ino=12768038556851320042 size=0 mode=-rwx------}} {0x1 fl=OpenDirectIO+OpenNonSeekable} source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa4 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @0 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa4] Write 131056 source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa5 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @131056 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa5] Write 131072 source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa6 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @262128 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa6] Write 16 source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa7 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @262144 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa7] Write 131056 source=fuse
[...]
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9c Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @267780080 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9c] Write 131072 source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9d Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @267911152 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9d] Write 16 source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9e Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @267911168 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9e] Write 131056 source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9f Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @268042224 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xc9f] Write 131072 source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca0 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @268173296 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca0] Write 16 source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca1 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268173312 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca1] Write 131056 source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca2 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @268304368 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca2] Write 131072 source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca3 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @268435440 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca3] Write 16 source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca4 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268435456 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:39+02:00] -> [ID=0xca4] Write error=EIO: Timeout when reading or writing data source=fuse
DEBU[2017-09-22T13:59:39+02:00] <- Write [ID=0xca5 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268435456 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca5] Write error=EIO: Timeout when reading or writing data source=fuse
DEBU[2017-09-22T13:59:54+02:00] <- Flush [ID=0xca6 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 fl=0x0 lk=0x5fc476094e3c05ce source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca6] Flush source=fuse
DEBU[2017-09-22T13:59:54+02:00] <- Release [ID=0xca7 Node=0x3 Uid=0 Gid=0 Pid=0] 0x1 fl=OpenWriteOnly rfl=0 owner=0x0 source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca7] Release source=fuse
Try setting connect_timeout=5m
like suggested in #100
Hei. I will try that, but doesn't that mean the underlying system has a slowness issue ?