Occasional crash under Windows - use after free?

Question

Occasional crash under Windows - use after free?

ncw opened this issue 3 years ago · comments

We've had several reports in the rclone forum of a crash which appears to be something to do with cgofuse. I dismissed these as some random stuff up with Windows, but we've had 3 reports now which is more than the co-incidence level!

These produced 3 very similar tracebacks

unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x40350e]
 
goroutine 17 [running, locked to thread]:
runtime.throw(0x1df5fc3, 0x5)
        runtime/panic.go:1117 +0x79 fp=0xc0007d7d08 sp=0xc0007d7cd8 pc=0x43d859
runtime.sigpanic()
        runtime/signal_windows.go:245 +0x2d6 fp=0xc0007d7d60 sp=0xc0007d7d08 pc=0x453796
indexbytebody(0x500000000000501, 0xaff, 0x470300, 0xc00006c300, 0xaff, 0x0, 0x500000000000501, 0x500000000000501, 0xaff, 0xc0007d7e08, ...)
        internal/bytealg/indexbyte_amd64.s:124 +0xce fp=0xc0007d7d68 sp=0xc0007d7d60 pc=0x40350e
runtime.findnull(0x500000000000501, 0xc0007d7e08)
        runtime/string.go:438 +0x87 fp=0xc0007d7dc0 sp=0xc0007d7d68 pc=0x458d67
runtime.gostring(0x500000000000501, 0xc0000ea8c0, 0xc00008e058)
        runtime/string.go:320 +0x36 fp=0xc0007d7e18 sp=0xc0007d7dc0 pc=0x471a56
github.com/billziss-gh/cgofuse/fuse._Cfunc_GoString(...)
        _cgo_gotypes.go:320
github.com/billziss-gh/cgofuse/fuse.c_GoString(...)
        github.com/billziss-gh/cgofuse@v1.5.0/fuse/host_cgo.go:708
github.com/billziss-gh/cgofuse/fuse.hostRelease(0x500000000000501, 0x2e86fc18, 0x0)
        github.com/billziss-gh/cgofuse@v1.5.0/fuse/host.go:284 +0x94 fp=0xc0007d7e78 sp=0xc0007d7e18 pc=0x17293b4
github.com/billziss-gh/cgofuse/fuse.go_hostRelease(...)
        github.com/billziss-gh/cgofuse@v1.5.0/fuse/host_cgo.go:926
_cgoexp_12ef5be0dd8c_go_hostRelease(0x2e86fb80)
        _cgo_gotypes.go:768 +0x3e fp=0xc0007d7ea0 sp=0xc0007d7e78 pc=0x172f8fe
runtime.cgocallbackg1(0x172f8c0, 0x2e86fb80, 0x0)
        runtime/cgocall.go:292 +0x19a fp=0xc0007d7f40 sp=0xc0007d7ea0 pc=0x40517a
runtime.cgocallbackg(0x172f8c0, 0x2e86fb80, 0x0)
        runtime/cgocall.go:228 +0xfc fp=0xc0007d7fb8 sp=0xc0007d7f40 pc=0x404edc
runtime.cgocallback(0x0, 0x0, 0x0)
        runtime/asm_amd64.s:788 +0xc0 fp=0xc0007d7fe0 sp=0xc0007d7fb8 pc=0x476180
runtime.goexit()
        runtime/asm_amd64.s:1371 +0x1 fp=0xc0007d7fe8 sp=0xc0007d7fe0 pc=0x476461

unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x40350e]
 
goroutine 147 [running, locked to thread]:
runtime.throw(0x1df5fc3, 0x5)
	runtime/panic.go:1117 +0x79 fp=0xc00071dd08 sp=0xc00071dcd8 pc=0x43d859
runtime.sigpanic()
	runtime/signal_windows.go:245 +0x2d6 fp=0xc00071dd60 sp=0xc00071dd08 pc=0x453796
indexbytebody(0x2f736569766f4d2f, 0x2d1, 0x470300, 0xc000645800, 0x2d1, 0x0, 0x2f736569766f4d2f, 0x2f736569766f4d2f, 0x2d1, 0xc00071de08, ...)
	internal/bytealg/indexbyte_amd64.s:124 +0xce fp=0xc00071dd68 sp=0xc00071dd60 pc=0x40350e
runtime.findnull(0x2f736569766f4d2f, 0xc00071de08)
	runtime/string.go:438 +0x87 fp=0xc00071ddc0 sp=0xc00071dd68 pc=0x458d67
runtime.gostring(0x2f736569766f4d2f, 0xc0003dba40, 0xc0000de058)
	runtime/string.go:320 +0x36 fp=0xc00071de18 sp=0xc00071ddc0 pc=0x471a56
github.com/billziss-gh/cgofuse/fuse._Cfunc_GoString(...)
	_cgo_gotypes.go:320
github.com/billziss-gh/cgofuse/fuse.c_GoString(...)
	github.com/billziss-gh/cgofuse@v1.5.0/fuse/host_cgo.go:708
github.com/billziss-gh/cgofuse/fuse.hostRelease(0x2f736569766f4d2f, 0x2ec2fc18, 0x0)
	github.com/billziss-gh/cgofuse@v1.5.0/fuse/host.go:284 +0x94 fp=0xc00071de78 sp=0xc00071de18 pc=0x17293b4
github.com/billziss-gh/cgofuse/fuse.go_hostRelease(...)
	github.com/billziss-gh/cgofuse@v1.5.0/fuse/host_cgo.go:926
_cgoexp_12ef5be0dd8c_go_hostRelease(0x2ec2fb80)
	_cgo_gotypes.go:768 +0x3e fp=0xc00071dea0 sp=0xc00071de78 pc=0x172f8fe
runtime.cgocallbackg1(0x172f8c0, 0x2ec2fb80, 0x0)
	runtime/cgocall.go:292 +0x19a fp=0xc00071df40 sp=0xc00071dea0 pc=0x40517a
runtime.cgocallbackg(0x172f8c0, 0x2ec2fb80, 0x0)
	runtime/cgocall.go:228 +0xfc fp=0xc00071dfb8 sp=0xc00071df40 pc=0x404edc
runtime.cgocallback(0x0, 0x0, 0x0)
	runtime/asm_amd64.s:788 +0xc0 fp=0xc00071dfe0 sp=0xc00071dfb8 pc=0x476180
runtime.goexit()
	runtime/asm_amd64.s:1371 +0x1 fp=0xc00071dfe8 sp=0xc00071dfe0 pc=0x476461

unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x4c360e]
 
goroutine 66 [running, locked to thread]:
runtime.throw(0x1bd0b9e, 0x5)
        runtime/panic.go:1116 +0x79 fp=0xc0003ffd20 sp=0xc0003ffcf0 pc=0x4fc259
runtime.sigpanic()
        runtime/signal_windows.go:249 +0x24f fp=0xc0003ffd50 sp=0xc0003ffd20 pc=0x510f4f
indexbytebody(0x500000000000501, 0xaff, 0x52cf00, 0xc000085380, 0xaff, 0x0, 0x500000000000501, 0x500000000000501, 0xaff, 0xc0003ffdf8, ...)
        internal/bytealg/indexbyte_amd64.s:124 +0xce fp=0xc0003ffd58 sp=0xc0003ffd50 pc=0x4c360e
runtime.findnull(0x500000000000501, 0xc0003ffdf8)
        runtime/string.go:454 +0x87 fp=0xc0003ffdb0 sp=0xc0003ffd58 pc=0x516627
runtime.gostring(0x500000000000501, 0xc000185040, 0xeb627ffa10)
        runtime/string.go:320 +0x36 fp=0xc0003ffe08 sp=0xc0003ffdb0 pc=0x52e616
github.com/billziss-gh/cgofuse/fuse._Cfunc_GoString(...)
        _cgo_gotypes.go:322
github.com/billziss-gh/cgofuse/fuse.c_GoString(...)
        github.com/billziss-gh/cgofuse@v1.4.0/fuse/host_cgo.go:686
github.com/billziss-gh/cgofuse/fuse.hostRelease(0x500000000000501, 0xeb627ffb40, 0x21000000000)
        github.com/billziss-gh/cgofuse@v1.4.0/fuse/host.go:284 +0x94 fp=0xc0003ffe68 sp=0xc0003ffe08 pc=0x15d0c54
github.com/billziss-gh/cgofuse/fuse.go_hostRelease(...)
        github.com/billziss-gh/cgofuse@v1.4.0/fuse/host_cgo.go:901
github.com/billziss-gh/cgofuse/fuse._cgoexpwrap_e4f666a8d790_go_hostRelease(0x500000000000501, 0xeb627ffb40, 0x0)
        _cgo_gotypes.go:802 +0x3c fp=0xc0003ffe90 sp=0xc0003ffe68 pc=0x15d641c
runtime.call32(0x0, 0xeb627ff930, 0xeb627ffaa0, 0x18)
        runtime/asm_amd64.s:540 +0x45 fp=0xc0003ffec0 sp=0xc0003ffe90 pc=0x5311e5
runtime.cgocallbackg1(0x0)
        runtime/cgocall.go:332 +0x1b8 fp=0xc0003fff58 sp=0xc0003ffec0 pc=0x4c51b8
runtime.cgocallbackg(0x0)
        runtime/cgocall.go:207 +0xe5 fp=0xc0003fffc0 sp=0xc0003fff58 pc=0x4c4f45
runtime.cgocallback_gofunc(0x0, 0x20, 0x2030001, 0x2030001)
        runtime/asm_amd64.s:794 +0xb2 fp=0xc0003fffe0 sp=0xc0003fffc0 pc=0x532992
runtime.goexit()
        runtime/asm_amd64.s:1374 +0x1 fp=0xc0003fffe8 sp=0xc0003fffe0 pc=0x532c61

The middle one of those reports has a mad pointer of 0x2f736569766f4d2f which decodes in ASCII as /seivoM/ which is highly suggestive of a use after free.

The other two have the pointer 0x500000000000501 which could be valid but looks a little fishy too.

If you check the forum posts I linked above then you'll see a bit more analysis.

So I think this is probably a use after free due to handling of Go strings, but I'm not certain!

Any help much appreciated.

Bill Zissimopoulos · Answer 1 · Thu Jun 10 2021 01:12:08 GMT+0800 (China Standard Time)

Given that cgofuse is just a thin layer over the underlying FUSE library this may actually be a problem with WinFsp-FUSE (i.e. the FUSE layer of WinFsp). I note that I do not know of any file system (and certainly not any of the test file systems) that uses the path argument of release so this is why it may have gone unnoticed so far.

Looking at the WinFsp-FUSE code in fuse_intf.c I can only find two locations where release is being called:

fsp_fuse_intf_Close - This happens when the kernel closes a file. I do not see anything suspicious here.
fsp_fuse_intf_Create - This happens when the kernel wants to create a file. If the file creation fails after the file has been opened (e.g. because a chown failed) the opened file must be closed using release. It looks like I messed up in this case and pass filedesc->PosixPath which has not been initialized yet. I believe I should be passing contexthdr->PosixPath (and indeed that is what the rest of this function uses). Sorry for that.

I will add a commit to WinFsp to fix this problem.

Bill Zissimopoulos · Answer 2 · Thu Jun 10 2021 01:29:09 GMT+0800 (China Standard Time)

I have added the commit referenced above, which should hopefully fix this problem.

Nick Craig-Wood · Answer 3 · Fri Jun 11 2021 06:27:19 GMT+0800 (China Standard Time)

Given that cgofuse is just a thin layer over the underlying FUSE library this may actually be a problem with WinFsp-FUSE (i.e. the FUSE layer of WinFsp). I note that I do not know of any file system (and certainly not any of the test file systems) that uses the path argument of release so this is why it may have gone unnoticed so far.

Ah. Rclone does everything with paths and almost complete ignores inodes which is not the usual way of building file systems.

Looking at the WinFsp-FUSE code in fuse_intf.c I can only find two locations where release is being called:

fsp_fuse_intf_Close - This happens when the kernel closes a file. I do not see anything suspicious here.

fsp_fuse_intf_Create - This happens when the kernel wants to create a file. If the file creation fails after the file has been opened (e.g. because a chown failed) the opened file must be closed using release.

That explains why it doesn't happen very often as it needs an additional error in create to trigger it.

It looks like I messed up in this case and pass filedesc->PosixPath which has not been initialized yet. I believe I should be passing contexthdr->PosixPath (and indeed that is what the rest of this function uses). Sorry for that.

No apologies necessary!

I have added the commit referenced above, which should hopefully fix this problem.

Excellent - thank you Bill :-)

Would it be possible for you to make a test build of WinFSP with this fix in so I can get the original poster to try it?

(I should do this myself really but I don't feel confident building WinFSP - my Windows VM doesn't have any of the MS dev tools on it and I don't know how to use them!)

Bill Zissimopoulos · Answer 4 · Fri Jun 11 2021 11:33:48 GMT+0800 (China Standard Time)

Would it be possible for you to make a test build of WinFSP with this fix in so I can get the original poster to try it?

I will see if I can produce a new Beta build during the weekend. I am in the process of moving and may not be able to do so timely.

(I should do this myself really but I don't feel confident building WinFSP - my Windows VM doesn't have any of the MS dev tools on it and I don't know how to use them!)

Building WinFsp is quite easy, but the problem is that the final build product needs to be signed. Windows will not run unsigned drivers unless you boot in "testsigning" mode. So even if you could build it, it might not be very useful.

Nick Craig-Wood · Answer 5 · Fri Jun 11 2021 22:58:26 GMT+0800 (China Standard Time)

Thanks Bill and no hurry :-) I hope the move goes well.

Bill Zissimopoulos · Answer 6 · Mon Jun 14 2021 04:48:26 GMT+0800 (China Standard Time)

I have added a new WinFsp release which should hopefully fix the issue:

https://github.com/billziss-gh/winfsp/releases/tag/v1.10B2

Please download this latest release and let me know if it fixes the problem.

Nick Craig-Wood · Answer 7 · Mon Jun 14 2021 19:16:04 GMT+0800 (China Standard Time)

I've given the people who had this problem the above link and I'll report back here on their testing.

Bill Zissimopoulos · Answer 8 · Thu Jul 01 2021 16:20:42 GMT+0800 (China Standard Time)

@ncw do you know if the latest WinFsp release has fixed the problem?

Nick Craig-Wood · Answer 9 · Thu Nov 18 2021 23:18:31 GMT+0800 (China Standard Time)

As far as I know it has fixed the problem - no more reports of it in the forum or the rclone bug tracker so I'll close this now.

Thank you :-)