GCSFUSE hanging while trying to write large files (~6GB)
tarun360 opened this issue · comments
Describe the issue
I am trying to cp a relatively large single file, around 6GB, through GCSFUSE, and it just hangs forever. I am able to use gsutil/gcloud-storage-rsync to copy the same file, so I am sure there's no bandwidth problem here. I have also ensured that there's enough space in --temp-dir=/tmp to write a 6GB file. Doing ls from separate terminal is also stuck.
I don't see this issue with smaller files around 200MB.
I think this issue is similar to #715.
This is greatly limiting our use of GCSFUSE, hence requesting to please have look urgently.
To Collect more Debug logs
Relevant GCSFUSE logs while doing cp:
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002906 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5228199936, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":109021661}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f0 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":109514277}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f2 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":109968981}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002908 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5229248512, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":110087270}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290a connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5230297088, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":110350364}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f4 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":110457680}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290c connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5231345664, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":110798379}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f8 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112676976}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f6 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112803534}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028fa connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112841686}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028fc connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112869553}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290e connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5232394240, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":112983238}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002910 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5233442816, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":113224758}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002912 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5234491392, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":113465212}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002914 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5235539968, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":113721204}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028fe connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":114112281}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002900 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":114194980}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002916 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5236588544, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":114386752}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002918 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5237637120, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":114636800}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002902 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":114689493}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291a connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5238685696, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":114994381}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002904 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":115484631}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291c connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5239734272, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":115805222}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002906 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":116074374}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291e connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5240782848, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":116348745}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002908 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":116654494}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002920 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5241831424, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":116973897}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290a connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":117296953}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290c connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":117931049}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002914 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":118536042}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290e connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":119144253}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002910 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":119719868}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002912 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":120347024}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002916 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":120978491}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002918 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":121618115}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291a connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":122236082}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291c connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":122920430}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002920 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":123971391}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291e connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":124041422}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002922 connection.go:416] \u003c- FlushFile (inode 4, PID 1809725)\n","timestampSeconds":1702017866,"timestampNanos":124142730}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req 0x9: \u003c- StatObject(\".bfuse/proj/scratch/gupttaru/file0.txt\")\n","timestampSeconds":1702017866,"timestampNanos":127842104}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req 0x9: -\u003e StatObject(\".bfuse/proj/scratch/gupttaru/file0.txt\") (140.90789ms): OK\n","timestampSeconds":1702017866,"timestampNanos":268764912}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req 0xa: \u003c- CreateObject(\".bfuse/proj/scratch/gupttaru/file0.txt\")\n","timestampSeconds":1702017866,"timestampNanos":268833249}
The logs stop being generated at this point, i.e. at CreateObject, even though the cp command hasn't been completed. On the bucket, I just a see an object with 0 bytes present.
Doing ls from a separate terminal is also stuck, and the last GCSFUSE log statement is the following:
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002954 connection.go:416] \u003c- LookUpInode (parent 3, name \"file0.txt\", PID 1820773)\n","timestampSeconds":1702018841,"timestampNanos":720568546}
System (please complete the following information):
- OS: RHEL8 (version 8.9)
- Platform: On-premise machine.
- Version: gcsfuse version 1.1.0 (Go version go1.20.4)
Hi @tarun360,
When dealing with large files, GCSFuse may require a significant amount of time to complete the upload. For instance, writing a 6GB file can take around 15 minutes. During this process, ls calls may appear to be stuck, but other read/write operations within the directory should remain functional.
Could you please confirm if GCSFuse is truly stuck or simply experiencing a longer upload duration due to the file size?
Thanks!
@ashmeenkaur Thanks for the reply.
Could you please confirm if GCSFuse is truly stuck or simply experiencing a longer upload duration due to the file size?
Indeed, I think its truly stuck. I left it overnight to check how much time it would take, however, even after 12 hours it was still stuck.
writing a 6GB file can take around 15 minutes.
Uploading the same file via gsutil takes less than 1 minute (without using any type of parallelism like parallel composite uploads). Is it expected, that a file which takes less than 1 minute to upload by gsutil CLI, will take 15 minutes on writing via GCSFUSE? Could you please provide more details on this, as I wasn't aware of this performance difference, and I couldn't find any reference to this in the documentation. Likewise, are reads from GCSFUSE expected to be equally slow as compared to downloading files via gsutil?
@tarun360 we don't have clear performance comparison with gsutil right now.
Coming to the issue, we are not able to reproduce this at our end. It would be easier for us to debug if you could provide more information like bucket_name. This will help us in getting the GCS server logs.
I would suggest:
(1) Try reproducing the issue with the latest version of gcsfuse.
(2) Please provide bucket_name via creating an internal customer support ticket.
Regards,
Prince Kumar.
@tarun360 Gentle reminder!
Could you please create a support ticket - https://support.cloud.google.com/portal/ ?
Thanks!
Hi @raj-prince , thanks for the reply.
I will update here after logging a support ticket. The project in which the bucket is doesn't allow logging a support ticket, so I need to get it changed via cloud admin. I am trying to get this expedited.
Hi @raj-prince , @ashmeenkaur
I tried the same operation today, and for some reason the behavior is different. Instead of getting stuck, it throws some timeout error:
WriteFile (inode 4, PID 0, handle 0, offset 5232394240, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6189 85658}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e2 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5233442816, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6194 70316}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e0 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":619510992}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e4 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5234491392, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6201 00741}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e2 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":620195750}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e4 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":620580663}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e6 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5235539968, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6207 65955}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e8 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5236588544, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6212 44026}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e6 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":621300854}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ea connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5237637120, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6218 60159}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e8 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":621883111}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ea connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":622295194}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ec connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5238685696, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6223 87506}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ec connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":622860002}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ee connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5239734272, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6229 93387}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f0 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5240782848, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6234 35009}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ee connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":623473372}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f2 connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5241831424, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6240 33831}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f4 connection.go:416] \u003c- SetInodeAttributes (inode 4, PID 2649659, mtime 2023-12-13 06:42:02.60670602 -0500 EST)\n","timestampSeconds":1702467722,"tim estampNanos":624092436}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f0 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":624123551}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f2 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":624559145}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f4 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702467722,"timestampNanos":624610152}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f6 connection.go:416] \u003c- FlushFile (inode 4, PID 2649659)\n","timestampSeconds":1702467722,"timestampNanos":624702607}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Starting a garbage collection run.\n","timestampSeconds":1702468306,"timestampNanos":807542532}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 132.741948ms.\n","timestampSeconds":1702468306,"timestampNanos":940426931}
{"name":"root","levelname":"ERROR","severity":"ERROR","message":"FlushFile: connection timed out, FileInode.Sync: SyncObject: Create: CreateObject: error in io.Copy: Post \"https://storage.googleapis.com/upload/storage/v1/b/<TRIMMED BY ME>": write tcp <TRIMMED BY ME>-\u003e<TRIMMED BY ME>: write: connection timed out\n","timestampSeconds":1702468360,"timestampNanos":820447859}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f6 connection.go:500] -\u003e Error: \"connection timed out\"\n","timestampSeconds":1702468360,"timestampNanos":820604536}
{"name":"root","levelname":"ERROR","severity":"ERROR","message":"fuse: *fuseops.FlushFileOp error: connection timed out\n","timestampSeconds":1702468360,"timestampNanos":820630051}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f8 connection.go:416] \u003c- ReleaseFileHandle (PID 0)\n","timestampSeconds":1702468360,"timestampNanos":820850891}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f8 connection.go:498] -\u003e OK ()\n","timestampSeconds":1702468360,"timestampNanos":820984966}
$ time cp /path1/file0.txt /path2/file0.txt
cp: failed to close '/path2/file0.txt': Connection timed out
real 10m9.015s
user 0m0.011s
sys 0m5.883s
The above error is again happening ONLY for relatively large files. The one I tried above is ~5GB.
The closest issue I found slightly resembling the above error is this, although this issue is marked as fixed.
Given this new finding, do you have any ideas about what could be going wrong with my setup here?
All these error signals some issue with internet connection. As we have tried this our side, which is working fine.
Could you please check VM's internet connection and try again?
Also, I haven't received any internal ticket yet. Did you file the ticket?
Thanks,
Prince Kumar.
Could you please check VM's internet connection and try again?
The network connection is fine since I am able to read/write the same file using gsutil.
Also, I am trying to read/write from a on-prem machine to bucket, not from a cloud VM. The network connection between on-prem machine and GCP seems fine, since I am able to read/write the same file using gsutil.
Also, I haven't received any internal ticket yet. Did you file the ticket?
Not yet. Due to holiday season, approvers are on leave. I'll update here once this is done.
Thanks,
Tarun
I created Google Cloud Support #48565270: GCSFUSE hanging while trying to write large files (~6GB)
Thanks for reaching out via Google cloud support. Further communication can be done via the support.
Is this issue always reproducible, @tarun360?
@ashmeenkaur Yes, its always reproducible.
Hi, I have added some more details on the support ticket, which I believe could be helpful in debugging this issue. I hope you have access to the support ticket.
The problem seems related to perhaps GCSFUSE and proxy server.
When creating a storage client in this function, in Lines 48-58 we are adding option.WithHTTPClient, so this means that the environment variable https_proxy will not be respected directly by the StorageClient created in line 71 by default.
For that, perhaps in the CreateHTTPClient function, we probably need something like
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
// other stuff ...
}
or clone the default transport and set your options in that transport (ref: https://stackoverflow.com/a/51848441/13416549).
I may be wrong here since I don't know the whole code, but am I missing anything in above logic?
After making the above change, I built gscfuse from source code, and it seems that the above issue I was facing has gone away.
After making the above change, I built
gscfusefrom source code, and it seems that the above issue I was facing has gone away.
Thanks for looking into this @tarun360. Did it also solve the 6GB file upload issue or just write: connection timed out issue?
Did it also solve the 6GB file upload issue or just write: connection timed out issue?
Without the above change, the data upload traffic wasn't being routed via the proxy server (verified by checking the proxy-server logs), so I suppose that might be related to some sort of rate-limiting by the firewalls for public endpoints, which lead to write: connection timed out.
With the above change, I am able to upload the 6GB file, so that probably implies that data is being routed via the proxy server (although I am yet to check proxy logs, but I am pretty sure that is the case since there's no other explaination of why this change would fix this issue).
Hi @tarun360,
GCSFuse v1.4.1 fixes this issue. Please feel free to re-open incase you still run into any failure.
Thanks