Mac OS X support

Question

Mac OS X support

rfjakob opened this issue 9 years ago · comments

Go support Mac OS X, as does the FUSE library we use, go-fuse.

gocrypts may actually work out of the box on OSX, but there probably are small issues that have to be sorted out.

At the very least, gocryptfs has to be tested on OSX. As I do not have a Mac to test on, this would be an opportunity for somebody from the Mac community to step up. Please comment here if you are interested.

Hasit Mistry · Answer 1 · Wed Jan 06 2016 17:28:47 GMT+0800 (China Standard Time)

I am interested in helping on this as much as I can. I will try to run it and comment back with what I find.

rfjakob · Answer 2 · Wed Jan 06 2016 21:21:59 GMT+0800 (China Standard Time)

Hi @hasit, great to hear that!

Siyuan Ren · Answer 3 · Thu Jan 07 2016 02:03:07 GMT+0800 (China Standard Time)

One thing it might have omitted is encryption of extended attributes. OS X makes extensive usage of extended attributes. For example, when you download a file with Google Chrome or Safari, the file will be marked with its original URL. When I tried EncFS before, the encrypted file always had the same extended attributes as the decrypted one, and thereby the plaintext could be easily obtained. I suspect the encryption of extended attribute will never cross the mind of regular Linux folk.

rfjakob · Answer 4 · Thu Jan 07 2016 02:12:28 GMT+0800 (China Standard Time)

Extended attributes are not implemented yet in gocryptfs. I don't know how big a problem that is on OSX, but at least we cannot leak data ;)

spaghetti2514 · Answer 5 · Thu Jan 07 2016 07:25:03 GMT+0800 (China Standard Time)

Attempting to install on OS X 10.11.2 with go get github.com/rfjakob/gocryptfs, I get the following

# github.com/rfjakob/gocryptfs/cryptfs
.go/src/github.com/rfjakob/gocryptfs/cryptfs/openssl_aead.go:35: undefined: openssl.NewGCMEncryptionCipherCtx
.go/src/github.com/rfjakob/gocryptfs/cryptfs/openssl_aead.go:76: undefined: openssl.NewGCMDecryptionCipherCtx
# github.com/hanwen/go-fuse/fuse/nodefs
.go/src/github.com/hanwen/go-fuse/fuse/nodefs/files_darwin.go:5: imported and not used: "time"
.go/src/github.com/hanwen/go-fuse/fuse/nodefs/syscall.go:11: undefined: syscall.SYS_UTIMENSAT

rfjakob · Answer 6 · Thu Jan 07 2016 08:28:40 GMT+0800 (China Standard Time)

@spaghetti2514 : Interesting. Can you run "openssl version" and post the output?

spaghetti2514 · Answer 7 · Thu Jan 07 2016 08:32:13 GMT+0800 (China Standard Time)

@rfjakob OpenSSL 1.0.2e 3 Dec 2015

spaghetti2514 · Answer 8 · Thu Jan 07 2016 08:34:36 GMT+0800 (China Standard Time)

I should point out that this probably has more to do with
https://github.com/spacemonkeygo/openssl/blob/master/ciphers_gcm.go#L15
than my openssl version

Removing the ",!darwin" part of that line eliminates those two errors for me

rfjakob · Answer 9 · Thu Jan 07 2016 08:43:09 GMT+0800 (China Standard Time)

I guess the "!darwin" is because openssl was too old in an earlier OSX
version. But 1.0.2e is very recent, can you file a bug against
spacemonkeygo/openssl?

On Thu, Jan 7, 2016 at 1:34 AM, spaghetti2514 notifications@github.com
wrote:

I should point out that this probably has more to do with
https://github.com/spacemonkeygo/openssl/blob/master/ciphers_gcm.go#L15
than my openssl version

Removing the ",!darwin" part of that line eliminates those two errors for
me

—
Reply to this email directly or view it on GitHub
#15 (comment).

spaghetti2514 · Answer 10 · Thu Jan 07 2016 08:55:41 GMT+0800 (China Standard Time)

1.0.2e is very recent because I installed it very recently. It is not the native OS X openssl because the native OS X openssl no longer exists. Apple deprecated the use of openssl a long time ago in favor of their own crypto libs, and finally stopped shipping openssl completely in 10.11. OS X users that still build things against openssl tend to install it from homebrew, an unofficial package manager for OS X.

I am unsure if excluding OS X in the build process should be considered a bug, since the only reason it works for me is that I'm building openssl myself via homebrew.
On the other hand, there is no longer an official openssl shipped with OS X at all, so maybe that should be assumed.
Still want me to open a bug report?

rfjakob · Answer 11 · Thu Jan 07 2016 09:17:42 GMT+0800 (China Standard Time)

In that case, it makes no sense at all anymore to exclude gcm from the build. Yes please file a ticket. I will take care of the other two issues as I am afraid I have caused them.

spaghetti2514 · Answer 12 · Thu Jan 07 2016 09:23:26 GMT+0800 (China Standard Time)

I have filed a ticket.

What other two issues are you referring to? just the unused time import and SYS_UTIMENSAT problems in go-fuse?

I think we'll run into other problems after that, as commenting some of that out to try to get farther along in the build reveals other build errors, but I can't be certain that they're not being caused by the way I'm stepping around previous errors.
Would you like me to wait until you've properly fixed the SYS_UTIMENSAT issue before checking for other errors?

rfjakob · Answer 13 · Thu Jan 07 2016 09:26:56 GMT+0800 (China Standard Time)

Yes, just the two in go-fuse. Commenting them out should be fine. Later
build problems are probably real, what do you get?
On Jan 7, 2016 02:23, "spaghetti2514" notifications@github.com wrote:

I have filed a ticket.

What other two issues are you referring to? just the unused time import
and SYS_UTIMENSAT problems in go-fuse?

I think we'll run into other problems after that, as commenting some of
that out to try to get farther along in the build reveals other build
errors, but I can't be certain that they're not being caused by the way I'm
stepping around previous errors.
Would you like me to wait until you've properly fixed the SYS_UTIMENSAT
issue before checking for other errors?

—
Reply to this email directly or view it on GitHub
#15 (comment).

spaghetti2514 · Answer 14 · Thu Jan 07 2016 09:30:57 GMT+0800 (China Standard Time)

After bypassing SYS_UTIMENSAT in syscall.go I get

# github.com/hanwen/go-fuse/fuse/pathfs
.go/src/github.com/hanwen/go-fuse/fuse/pathfs/loopback.go:140: undefined: sysUtimensat
.go/src/github.com/hanwen/go-fuse/fuse/pathfs/loopback.go:140: undefined: _AT_SYMLINK_NOFOLLOW

After commenting out line 140 in loopback.go and changing the return err to return 0, I get

# github.com/rfjakob/gocryptfs/pathfs_frontend
.go/src/github.com/rfjakob/gocryptfs/pathfs_frontend/file.go:213: undefined: syscall.Fallocate

After replacing line 213 in file.go with err = syscall.EINTR, the build succeeds.

rfjakob · Answer 15 · Fri Jan 08 2016 05:35:14 GMT+0800 (China Standard Time)

Ok, I hope I have fixed all of this, but I do not have a Mac to test on. @spaghetti2514 , can you

"git pull" in gocryptfs
Replace hanwen/go-fuse with my branch that has the OSX build fix: https://github.com/rfjakob/go-fuse . I will get it upstream once you confirm it fixes the issue (and works).

spaghetti2514 · Answer 16 · Sat Jan 09 2016 06:50:17 GMT+0800 (China Standard Time)

@rfjakob

go-fuse/fuse/nodefs/files_darwin.go:74: cannot use int64(t.Nanosecond() / 1000) (type int64) as type int32 in assignment

Edit: bypassing that leads to

go-fuse/fuse/pathfs/loopback_darwin.go:44: syntax error: unexpected name, expecting )

rfjakob · Answer 17 · Sat Jan 09 2016 07:29:14 GMT+0800 (China Standard Time)

Thanks, should be fixed, please pull go-fuse.

spaghetti2514 · Answer 18 · Sat Jan 09 2016 11:16:51 GMT+0800 (China Standard Time)

# github.com/rfjakob/go-fuse/fuse/pathfs
.go/src/github.com/rfjakob/go-fuse/fuse/pathfs/loopback_darwin.go:33: undefined: time in time.Time
.go/src/github.com/rfjakob/go-fuse/fuse/pathfs/loopback_darwin.go:42: undefined: time in time.Time
.go/src/github.com/rfjakob/go-fuse/fuse/pathfs/loopback_darwin.go:56: undefined: f in f.lock
.go/src/github.com/rfjakob/go-fuse/fuse/pathfs/loopback_darwin.go:58: undefined: f in f.lock

spaghetti2514 · Answer 19 · Sat Jan 09 2016 11:43:52 GMT+0800 (China Standard Time)

Other various things

go-fuse/fuse/pathfs/loopback_darwin.go:35: cannot use int64(t.Nanosecond() / 1000) (type int64) as type int32 in assignment

gocryptfs/pathfs_frontend/compat_darwin.go:3: imported and not used: "syscall"

gocryptfs/pathfs_frontend/compat_darwin.go:13: missing return at end of function

rfjakob · Answer 20 · Sat Jan 09 2016 18:56:44 GMT+0800 (China Standard Time)

Ok, I need a cross compiler ;)
I will get back to you once I have it building.

rfjakob · Answer 21 · Sat Jan 09 2016 20:14:53 GMT+0800 (China Standard Time)

Turns out that for cross compilation to work, I would need the C header files from OSX, because go-fuse uses cgo. And to get those, I'd have to extract them from XCode on a running OSX machine. OMG.

@spaghetti2514 : The things you have reported should be fixed, please pull. If you still have the patience, I think we are done soon.

Hasit Mistry · Answer 22 · Sun Jan 10 2016 04:32:13 GMT+0800 (China Standard Time)

I can confirm the same error. Following is the output I get on my MacBook Pro (Retina, 13-inch, Late 2013) OSX El Capitan version 10.11.2 .

# github.com/hanwen/go-fuse/fuse/nodefs
go/src/github.com/hanwen/go-fuse/fuse/nodefs/files_darwin.go:5: imported and not used: "time"
go/src/github.com/hanwen/go-fuse/fuse/nodefs/syscall.go:11: undefined: syscall.SYS_UTIMENSAT
# github.com/spacemonkeygo/openssl
go/src/github.com/spacemonkeygo/openssl/bio.go:21:10: fatal error: 'openssl/bio.h' file not found
#include <openssl/bio.h>
         ^
1 error generated.

rfjakob · Answer 23 · Sun Jan 10 2016 04:37:18 GMT+0800 (China Standard Time)

Hasit, you need to replace go-fuse with my branch, and you need openssl installed. The errors you see are already fixed.

spaghetti2514 · Answer 24 · Sun Jan 10 2016 06:44:46 GMT+0800 (China Standard Time)

@rfjakob Build succeeds with latest changes

rfjakob · Answer 25 · Sun Jan 10 2016 07:07:50 GMT+0800 (China Standard Time)

Great. Pushed to upstream as hanwen/go-fuse#89 .

./test.bash ?

spaghetti2514 · Answer 26 · Sun Jan 10 2016 07:30:43 GMT+0800 (China Standard Time)

ok      github.com/rfjakob/gocryptfs/cryptfs    1.886s
gocryptfs v0.7.1; on-disk format 2
remove /tmp/gocryptfs_main_test//TestExampleFsV04: resource busy
FAIL    github.com/rfjakob/gocryptfs/integration_tests  0.055s

Edit: That was caused by running the script twice when the first one failed to unmount any of the temporary filesystems. After unmounting the filesystems in /tmp and trying again I got

ok      github.com/rfjakob/gocryptfs/cryptfs    1.828s
gocryptfs v0.7.1; on-disk format 2
--- FAIL: TestExampleFSv04 (0.96s)
    example_filesystems_test.go:21: open /tmp/gocryptfs_main_test/TestExampleFsV04/status.txt: no such file or directory
--- FAIL: TestExampleFSv05 (0.14s)
    example_filesystems_test.go:21: open /tmp/gocryptfs_main_test/TestExampleFsV05/status.txt: no such file or directory
--- FAIL: TestExampleFSv06 (0.12s)
    example_filesystems_test.go:21: open /tmp/gocryptfs_main_test/TestExampleFsV06/status.txt: no such file or directory
--- FAIL: TestExampleFSv06PlaintextNames (0.10s)
    example_filesystems_test.go:21: open /tmp/gocryptfs_main_test/TestExampleFsV06PlaintextNames/status.txt: no such file or directory
exec: "fusermount": executable file not found in $PATH
exit status 10
FAIL    github.com/rfjakob/gocryptfs/integration_tests  3.267s

fusermount is linux-specific by the way. Normal umount works fine for fuse filesystems on OS X

spaghetti2514 · Answer 27 · Wed Mar 30 2016 21:40:11 GMT+0800 (China Standard Time)

@rfjakob The OS X issue at spacemonkeygo/openssl has not been addressed at all in months.
Would you consider forking it, making the small change yourself, and having gocryptfs depend on your fork instead? Considering how infrequently changes are made to the repo, it shouldn't be much effort at all for you to "maintain" and would allow you to address issues like this much more efficiently.

rfjakob · Answer 28 · Sun Jul 03 2016 02:10:37 GMT+0800 (China Standard Time)

@spaghetti2514 gocryptfs v0.10, released a month ago, replaced the dependency to spacemonkeygo/openssl with our own minimal wrapper, stupidgcm. So this problem should be gone (but maybe we gained other ones ;) ).

The example filesystem failures look look pretty bad, though. Does gocryptfs work at all?

rfjakob · Answer 29 · Sun Jul 03 2016 02:11:11 GMT+0800 (China Standard Time)

BTW since commit d5b7eb3 I am calling umount on OSX.

spaghetti2514 · Answer 30 · Sun Jul 03 2016 23:35:50 GMT+0800 (China Standard Time)

# github.com/rfjakob/gocryptfs/internal/nametransform
.go/src/github.com/rfjakob/gocryptfs/internal/nametransform/diriv.go:38: undefined: syscall.Openat
.go/src/github.com/rfjakob/gocryptfs/internal/nametransform/longnames.go:68: undefined: syscall.Unlinkat
.go/src/github.com/rfjakob/gocryptfs/internal/nametransform/longnames.go:89: undefined: syscall.Openat

Edit: After bypassing the above:

# github.com/rfjakob/gocryptfs/internal/fusefrontend
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/file_allocate_truncate.go:62: undefined: syscall.Fallocate
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs.go:135: undefined: syscall.Openat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs.go:207: undefined: syscall.Mknodat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs.go:297: undefined: syscall.Unlinkat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs.go:410: undefined: syscall.Renameat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs.go:419: undefined: syscall.Renameat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs_dir.go:116: undefined: syscall.Openat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs_dir.go:138: undefined: syscall.Openat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs_dir.go:171: undefined: syscall.Renameat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs_dir.go:185: undefined: syscall.Renameat
.go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/fs_dir.go:185: too many errors

Edit Edit: Build succeeds after additionally bypassing the above. I'm not going to run the tests though as I'm pretty confident nothing will work with a commented out fallocate and a bunch of *at syscalls being replaced with their non-at version and with file descriptors removed

rfjakob · Answer 31 · Sun Jul 03 2016 23:42:34 GMT+0800 (China Standard Time)

Oh dear, OSX does not have the openat syscall. Fun times.

rfjakob · Answer 32 · Mon Jul 04 2016 01:20:32 GMT+0800 (China Standard Time)

This has Fallocate and Openat fixed: 9b725c1
Working on Renameat.

rfjakob · Answer 33 · Mon Jul 04 2016 01:24:23 GMT+0800 (China Standard Time)

But it broke Prealloc. Fixing in the next patch.

rfjakob · Answer 34 · Mon Jul 04 2016 02:24:46 GMT+0800 (China Standard Time)

Ok, I think I got everything. Please try latest master ( 741bf07 ).

spaghetti2514 · Answer 35 · Mon Jul 04 2016 10:33:53 GMT+0800 (China Standard Time)

Builds successfully though with the following:

# github.com/rfjakob/gocryptfs/internal/stupidgcm
.go/src/github.com/rfjakob/gocryptfs/internal/stupidgcm/locking.go:16:2: warning: indirection of non-volatile null pointer will be deleted, not trap [-Wnull-dereference]
.go/src/github.com/rfjakob/gocryptfs/internal/stupidgcm/locking.go:16:2: note: consider using __builtin_trap() or qualifying pointer with 'volatile'

Resulting binary seems to work but often prints open /proc/cpuinfo: no such file or directory

All of test.bash:

open /proc/cpuinfo: no such file or directory
gocryptfs v0.12-44-g77e7abd; go-fuse 7b28148; 2016-07-04 go1.6.2
?       github.com/rfjakob/gocryptfs    [no test files]
ok      github.com/rfjakob/gocryptfs/internal/configfile    2.406s
ok      github.com/rfjakob/gocryptfs/internal/contentenc    0.033s
ok      github.com/rfjakob/gocryptfs/internal/cryptocore    0.033s
?       github.com/rfjakob/gocryptfs/internal/fusefrontend  [no test files]
ok      github.com/rfjakob/gocryptfs/internal/nametransform 0.023s
ok      github.com/rfjakob/gocryptfs/internal/prefer_openssl    0.019s
ok      github.com/rfjakob/gocryptfs/internal/readpassword  0.168s
# github.com/rfjakob/gocryptfs/tests/test_helpers
tests/test_helpers/helpers.go:301: invalid operation: st.Blocks * st.Blksize (mismatched types int64 and int32)
# github.com/rfjakob/gocryptfs/internal/stupidgcm
internal/stupidgcm/locking.go:16:2: warning: indirection of non-volatile null pointer will be deleted, not trap [-Wnull-dereference]
internal/stupidgcm/locking.go:16:2: note: consider using __builtin_trap() or qualifying pointer with 'volatile'
ok      github.com/rfjakob/gocryptfs/internal/stupidgcm 1.434s
?       github.com/rfjakob/gocryptfs/internal/syscallcompat [no test files]
?       github.com/rfjakob/gocryptfs/internal/tlog  [no test files]
FAIL    github.com/rfjakob/gocryptfs/tests/example_filesystems [build failed]
FAIL    github.com/rfjakob/gocryptfs/tests/matrix [build failed]
FAIL    github.com/rfjakob/gocryptfs/tests/normal [build failed]
FAIL    github.com/rfjakob/gocryptfs/tests/plaintextnames [build failed]

rfjakob · Answer 36 · Mon Jul 04 2016 14:24:13 GMT+0800 (China Standard Time)

Does not look bad, but obviously most of the tests have not been run. I have fixed that in 9d17fdb and also got rid of the /proc/cpuinfo warnings.

spaghetti2514 · Answer 37 · Mon Jul 04 2016 15:08:36 GMT+0800 (China Standard Time)

Tests seem to be hanging somewhere in tests/example_filesystems
df shows currently mounted filesystems named v0.7, v0.7-plaintextnames.mnt, and v0.9 under one directory, and default-plain under another if that helps you diagnose how far the tests have progressed at all.

Oh wait, free inode count dropped for each of them after several minutes. I guess the tests are working, just very slowly. I'll post the full output tomorrow if it finishes.

Edit: after no progress in 10 hours I think it's safe to assume these aren't finishing. Is there a way to run tests more verbosely to see where it's hanging?

spaghetti2514 · Answer 38 · Sun Jul 10 2016 00:03:25 GMT+0800 (China Standard Time)

@rfjakob just for the record, you referenced this issue as "ticket #25" and "ticket #" in the 1.0-rc1 README update even though both links correctly go to this issue (#15)

rfjakob · Answer 39 · Sun Jul 10 2016 18:23:52 GMT+0800 (China Standard Time)

Fixed in the README, thanks!

You can run ./test.bash -v, this will at least show each test being run.

Also, sorry for replying so late - github does not send out notifications when a post is edited. I only noticed your edit about the hanging tests right now when replying to the README issues.

spaghetti2514 · Answer 40 · Mon Jul 11 2016 01:13:16 GMT+0800 (China Standard Time)

I don't think -v is helping. It's more verbose, but it's not printing any information about what's going on right before the hang

last line printed before hanging is

?       github.com/rfjakob/gocryptfs/internal/tlog    [no test files]

rfjakob · Answer 41 · Mon Jul 11 2016 16:55:09 GMT+0800 (China Standard Time)

Ok I guess it hangs in the Init() of the next test. How about

 ./test.bash -v -timeout 60s

? I hope this gets us a backtrace for the hung location.

rfjakob · Answer 42 · Mon Jul 11 2016 16:59:37 GMT+0800 (China Standard Time)

Or maybe not. This should have already triggered. Looks like it does affect
time spent in Init().

-timeout t If a test runs longer than t, panic. The default is 10

minutes (10m).

I'll add more debugging info tonight.

rfjakob · Answer 43 · Mon Jul 11 2016 17:01:04 GMT+0800 (China Standard Time)

"Looks like it does NOT affect time spent in Init()"

rfjakob · Answer 44 · Tue Jul 12 2016 02:43:48 GMT+0800 (China Standard Time)

@spaghetti2514 please try again, I have fixed a bug (the cleanup function was calling fusermount instead of umount on OSX) and added output for the Init function (actually it is called TestMain). Enable with -v.

spaghetti2514 · Answer 45 · Tue Jul 12 2016 10:33:24 GMT+0800 (China Standard Time)

Last output is still

?       github.com/rfjakob/gocryptfs/internal/tlog  [no test files]

before it hangs. ctrl+c after a while lets it continue (which I'm pretty sure it wasn't doing before) with

^Cexample_filesystems: testing with "-openssl=false"

Backtrace:

umount(/private/tmp/gocryptfs-test-parent/053481967/default-plain): Resource busy -- try 'diskutil unmount'
exit status 1
panic: exit status 1

goroutine 1 [running]:
panic(0x4255e80, 0xc82024cbe0)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3e6
github.com/rfjakob/gocryptfs/tests/test_helpers.UnmountPanic(0xc8200147c0, 0x32)
    ~/.go/src/github.com/rfjakob/gocryptfs/tests/test_helpers/helpers.go:153 +0x110
github.com/rfjakob/gocryptfs/tests/matrix.TestMain(0xc82004fef8)
    ~/.go/src/github.com/rfjakob/gocryptfs/tests/matrix/matrix_test.go:48 +0x628
main.main()
    github.com/rfjakob/gocryptfs/tests/matrix/_test/_testmain.go:78 +0x114
FAIL    github.com/rfjakob/gocryptfs/tests/matrix   105.947s

Romain · Answer 46 · Tue Oct 04 2016 20:16:01 GMT+0800 (China Standard Time)

I'm progressing on my journey to build on macOS from source, and just hit this one

⇒  ./build-without-openssl.bash
# github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse
internal/fusefrontend_reverse/rfs.go:138: cannot use st.Dev (type int32) as type uint64 in field value

It could be a red herring but Stat_t.Dev is documented as a unit64, but the macOS amd64 source says int32. I am building on go version go1.7.1 darwin/amd64.

Any ideas? The introduction of fusefrontend_reverse looks quite recent.

rfjakob · Answer 47 · Tue Oct 04 2016 20:33:45 GMT+0800 (China Standard Time)

Yes, fusefrontend_reverse is very new! You are the first one to build it on OSX ;)

We need to cast st.Dev to uint64, in other words, please replace "st.Dev" with
"uint64(st.Dev)" and see if anything else fails.

Romain · Answer 48 · Tue Oct 04 2016 20:36:17 GMT+0800 (China Standard Time)

Perfect, thanks for the quick reply! Everything builds fine now. I also got around it by changing type devIno struct to use int32 but that would break a few things on Linux :)

rfjakob · Answer 49 · Tue Oct 04 2016 20:43:21 GMT+0800 (China Standard Time)

Yes probably :)

But if you want to submit a pull request with the uint64 cast I will merge it right away.

Romain · Answer 50 · Wed Oct 05 2016 20:44:01 GMT+0800 (China Standard Time)

Making progress. Creating a folder works well, but mounting crashes with:

panic: darwin has no splice

This comes from fuse/splice_darwin.go which is not implemented. Is splice required at all for gocryptfs? Should I use a different version of Fuse for macOS (currently using 3.5.2)?

rfjakob · Answer 51 · Wed Oct 05 2016 20:58:10 GMT+0800 (China Standard Time)

No, should work without splice as well. Seems the go-fuse lib should check if splice is available before using it.

Joss Brown · Answer 52 · Thu Oct 27 2016 22:24:53 GMT+0800 (China Standard Time)

 ❯ go get github.com/rfjakob/gocryptfs
# github.com/rfjakob/gocryptfs/internal/stupidgcm
go/src/github.com/rfjakob/gocryptfs/internal/stupidgcm/locking.go:11:10: fatal error: 'openssl/evp.h' file not found
#include <openssl/evp.h>
         ^
1 error generated.
>>> elapsed time 2m13s

10.11.6

How can I go about to installing gocryptfs correctly?

rfjakob · Answer 53 · Thu Oct 27 2016 23:37:09 GMT+0800 (China Standard Time)

Looks like you don't have openssl installed - please try ./build-without-openssl.bash

Joss Brown · Answer 54 · Fri Oct 28 2016 03:14:30 GMT+0800 (China Standard Time)

openssl is actually native to macOS. I also have a homebrewed version of openssl, which is normally keg only: I just linked it to one of the bin directories, but gocryptfs didn't install either. (evp.h is actually present in the openssl include directory; so I don't know what the problem could be… have to continue trying.)

rfjakob · Answer 55 · Fri Oct 28 2016 05:05:17 GMT+0800 (China Standard Time)

I can't comment on that, but note that if your cpu has aes acceleration there is NO downside in building without openssl. (edit on Dec 10, 2016: the "NO" was missing)

Bailey Brown · Answer 56 · Fri Dec 02 2016 02:28:47 GMT+0800 (China Standard Time)

I have a Mac that I rarely use. I tried gocryptfs on it a while back, and it seemed to work fine then.

I upgraded it to the lastest OSX (Sierra) and tried gocryptfs on it again.

gocryptfs wouldn't compile (with go 1.7.3) because

# github.com/rfjakob/gocryptfs/internal/ctlsock
internal/ctlsock/ctlsock_serve.go:74: undefined: syscall.PathMax

So I changed syscall.PathMax to 4096.

        // 2*PATH_MAX is definitely big enough for requests to decrypt or
        // encrypt paths.
-       buf := make([]byte, 2*syscall.PathMax)
+       //buf := make([]byte, 2*syscall.PathMax)
+       buf := make([]byte, 2*4096)

It looks like syscall.PathMax is normally a const 0x1000 (4096) in go. https://golang.org/pkg/syscall/. On Ubuntu, PATH_MAX is 4096 in <linux/limits.h>.

I wonder why it's missing in OSX? In syslimits.h, OSX has PATH_MAX defined to 1024. But I think here you just want to make sure the buffer is big enough, so I think it wouldn't hurt to use 4096 there.

So I compiled it, but then I had this weird problem. When go-fuse tried to glob /dev/osxfuse*, it didn't find any fuse device files, so it said "no FUSE devices found".

I tried re-installing "FUSE for macOS" aka osxfuse, from both the project's installer and from brew. I tried disabling Apple SIP (System Integrity Protection) thinking that was preventing osxfuse from loading.

But that wasn't it.

Then I mounted something with encfs, and it worked. And then I could see the osxfuse files in /dev.

So then gocryptfs worked too. And turning SIP back on didn't make it not work, as long as encfs mounted something before gocryptfs did.

The problem seems to be that somebody needs to run

/Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse

before go-fuse tries to open a fuse device file.

I put logic in mount_darwin.go to, if there were no /dev/osxfuse* files, then run load_osxfuse and try again.

But then it couldn't open any of them. It said they were all busy.

So I made it run load_osxfuse, then, in a loop, try to open any, and if none were opened, then sleep 100ms and try again (50 times, so 5 secs). It still didn't work. I mean it never worked the first time I ran gocryptfs, but running gocryptfs again worked.

So then I put this code in main.go in gocryptfs to make it run load_osxfuse before it did anything else (with no changes to go-fuse)

And now it works (including the first time after a fresh boot).


+func loadFuseIfNeeded() {
+       if runtime.GOOS == "darwin" {
+               const oldLoadOsxFuseBin = "/Library/Filesystems/osxfusefs.fs/Support/load_osxfusefs"
+               const newLoadOsxFuseBin = "/Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse"
+               bin := oldLoadOsxFuseBin
+                if _, err := os.Stat(newLoadOsxFuseBin); err == nil {
+                        bin = newLoadOsxFuseBin
+                }
+                cmd := exec.Command(bin);
+                cmd.Run();
+       }
+}
+
 func main() {
+       loadFuseIfNeeded()

Bailey Brown · Answer 57 · Fri Dec 02 2016 04:09:21 GMT+0800 (China Standard Time)

I think this is maybe better (more directed). It works as well.


diff --git a/main.go b/main.go
index 8ab180e..a4a46f7 100644
--- a/main.go
+++ b/main.go
@@ -3,7 +3,7 @@ package main
 import (
        "fmt"
        "os"
-
+       "os/exec"
        "path/filepath"
        "runtime"
        "runtime/pprof"
@@ -118,12 +118,33 @@ func printVersion() {
                tlog.ProgramName, GitVersion, buildFlags, GitVersionFuse, built)
 }

+func loadFuseIfNeeded() {
+       if runtime.GOOS == "darwin" {
+               if _, err := os.Stat("/dev/osxfuse0"); err != nil {
+                       const oldLoadOsxFuseBin = "/Library/Filesystems/osxfusefs.fs/Support/load_osxfusefs"
+                       const newLoadOsxFuseBin = "/Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse"
+                       bin := oldLoadOsxFuseBin
+                       if _, err := os.Stat(newLoadOsxFuseBin); err == nil {
+                               bin = newLoadOsxFuseBin
+                       }
+                       cmd := exec.Command(bin);
+                       cmd.Run();
+               }
+       }
+}
+
 func main() {
        runtime.GOMAXPROCS(4)
        var err error
        // Parse all command-line options (i.e. arguments starting with "-")
        // into "args". Path arguments are parsed below.
        args := parseCliOpts()
+
+       // On some OSes we might need to load fuse if we are mounting a
+       // filesystem.
+       if (flagSet.NArg() == 2) {
+               loadFuseIfNeeded()
+       }
        // Fork a child into the background if "-fg" is not set AND we are mounting
        // a filesystem. The child will do all the work.
        if !args.fg && flagSet.NArg() == 2 {

rfjakob · Answer 58 · Wed Dec 07 2016 05:07:36 GMT+0800 (China Standard Time)

Nice detective work! When you run load_osxfuse manually, does it work as well?

spaghetti2514 · Answer 59 · Wed Dec 07 2016 09:53:11 GMT+0800 (China Standard Time)

When you run load_osxfuse manually, does it work as well?

After applying bailey27's fix for syscall.PathMax and then manually running load_osxfuse, gocryptfs works fine for me. I did not try it without manually running load_osxfuse to try and reproduce bailey27's issue, but assuming it's accurate, running manually fixes it.

Also of note, test.bash fails to run on OS X due to the flock utility not existing

openssl is actually native to macOS.

OS X ships an (old) openssl binary and libcrypto, but has not shipped openssl headers since before El Capitan. The headers are necessary to build against it. Even if you could build against it, it's too old to support some of the functions required by stupidgcm

Homebrew recently started refusing to link their modern openssl to system locations, citing concerns that software built against their headers would end up linked to OS X's old openssl. They now recommend explicitly passing include dirs for homebrew's openssl when building anything against it.

Edit: I am currently accomplishing this by adding

// #cgo LDFLAGS: -L/usr/local/opt/openssl/lib
// #cgo CFLAGS: -I/usr/local/opt/openssl/include

to stupidgcm.go

Bailey Brown · Answer 60 · Thu Dec 08 2016 05:03:13 GMT+0800 (China Standard Time)

@rfjakob,

Yes, it worked when I ran load_osxfuse manually before running gocryptfs.

@spaghetti2514 ,

I seem to be linking with the openssl that came with brew.


$ otool -L gocryptfs
gocryptfs:
	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

But I didn't have to do anything special with include or library paths to get it to compile.

spaghetti2514 · Answer 61 · Thu Dec 08 2016 07:47:30 GMT+0800 (China Standard Time)

@bailey27
Is there something in your environment allowing you to do that? Or maybe some old symlinks in a more system-wide location? Without include and library paths I get the same error about openssl/evp.h that @JayBrown got.

Bailey Brown · Answer 62 · Thu Dec 08 2016 14:52:09 GMT+0800 (China Standard Time)

@spaghetti2514 ,

I re-installed my mac from scratch about two years ago. I think that's maybe why I don't have any openssl besides what comes with brew (installed/linked into /usr/local).

I don't know why my go build environment is looking in /usr/local/include. I haven't set any environment variables about it. My go isn't installed from bew (I downloaded the mac installer from the golang.org).

It looks like when I compile something from the command line with cc it doesn't look for headers in /usr/local/include. So I think it isn't system wide.

rfjakob · Answer 63 · Sun Dec 11 2016 03:46:24 GMT+0800 (China Standard Time)

@bailey27 I have dropped syscall.PathMax and now just use 5000 hardcoded ( 12374be ). Unless Linux raises the 4096-byte limit, that should be good enough.

rfjakob · Answer 64 · Sun Dec 11 2016 04:17:50 GMT+0800 (China Standard Time)

@spaghetti2514 I guess virtually all Macs that are running today have AES acceleration, so I'd just completely disregard openssl and build using ./build-without-openssl.bash.

You can check the impact of using openssl by commenting out line 51 in benchmark.bash and enabling one of the other two lines below: https://github.com/rfjakob/gocryptfs/blob/master/benchmark.bash#L51

spaghetti2514 · Answer 65 · Sun Dec 11 2016 05:17:07 GMT+0800 (China Standard Time)

@rfjakob benchmark.bash

Testing gocryptfs at /tmp/benchmark.bash.fk2
md5sum: stat '/tmp/linux-3.0.tar.gz': No such file or directory
Downloading linux-3.0.tar.gz
/tmp/linux-3.0.tar.gz                 100%[========================================================================>]  92.20M  3.16MB/s    in 29s     
2016-12-10 15:14:43 URL:https://cdn.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.gz [96675825/96675825] -> "/tmp/linux-3.0.tar.gz" [1]
WRITE: dd: bs: illegal numeric value
rm: zero: No such file or directory
./benchmark.bash: line 62: fusermount: command not found

rfjakob · Answer 66 · Wed Dec 14 2016 05:19:25 GMT+0800 (China Standard Time)

@spaghetti2514 Fixed in 2bacbdf . Also, I have added proper option parsing, so you can now run

./benchmark.bash -openssl=false

or

./benchmark.bash -openssl=true

spaghetti2514 · Answer 67 · Wed Dec 14 2016 07:34:57 GMT+0800 (China Standard Time)

@rfjakob

$ ./benchmark.bash -openssl=false
Testing gocryptfs at /tmp/benchmark.bash.SzA
md5sum: stat '/tmp/linux-3.0.tar.gz': No such file or directory
Downloading linux-3.0.tar.gz
/tmp/linux-3.0.tar.gz                 100%[========================================================================>]  92.20M  2.95MB/s    in 32s     
2016-12-13 17:24:47 URL:https://cdn.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.gz [96675825/96675825] -> "/tmp/linux-3.0.tar.gz" [1]
WRITE: 262144000 bytes transferred in 14.460788 secs (18127919 bytes/sec)
UNTAR: /usr/bin/time: illegal option -- f
usage: time [-lp] command.
./benchmark.bash: line 76: fusermount: command not found

Removing the -f option from time:

$ ./benchmark.bash -openssl=false   
Testing gocryptfs at /tmp/benchmark.bash.gjd
WRITE: 262144000 bytes transferred in 16.244092 secs (16137806 bytes/sec)
UNTAR: linux-3.0/arch/microblaze/boot/dts/system.dts: Can't set permissions to 0755Can't update time for linux-3.0/arch/microblaze/boot/dts/system.dts
linux-3.0/sound/soc/davinci/Makefile: Can't create 'linux-3.0/sound/soc/davinci/Makefile'
linux-3.0/sound/soc/davinci/davinci-evm.c: Can't create 'linux-3.0/sound/soc/davinci/davinci-evm.c'
linux-3.0/sound/soc/davinci/davinci-i2s.c: Can't create 'linux-3.0/sound/soc/davinci/davinci-i2s.c'
[snip 694 lines]
linux-3.0/virt/kvm/irq_comm.c: Can't create 'linux-3.0/virt/kvm/irq_comm.c'
linux-3.0/virt/kvm/kvm_main.c: Can't create 'linux-3.0/virt/kvm/kvm_main.c'
tar: Error exit delayed from previous errors.
      245.72 real         3.28 user        12.95 sys
./benchmark.bash: line 76: fusermount: command not found

There was a very long pause after the first permission/utime error, and then the flood of create errors all came at once.

mhogomchungu · Answer 68 · Sun Jan 29 2017 03:27:56 GMT+0800 (China Standard Time)

Greeting to all who have participated in this discussion.

I have a project called "SiriKali"[1] and i recently added support for OSX and would appreciate if somebody here who have gocryptfs running in OSX would try it out and report if it works just fine with gocryptfs. Git versions of both gocryptfs and SiriKali will have to be used for this test.

[1] https://github.com/mhogomchungu/sirikali

Paul Sarena · Answer 69 · Sat Feb 11 2017 16:38:04 GMT+0800 (China Standard Time)

At first look, gocryptfs appears to be working for me on OS X El Capitan 10.11.6. I'm able to init, mount, write, unmount, mount, read and get back what was written. There is still work to be done to pass all the tests.

Here are some notes:

Build and run gocryptfs on Mac

Build gocryptfs "without openssl" on OS X / mac OS

Install go from https://golang.org/dl/ -or- using MacPorts: port install go -or- Homebrew: brew install go. I used MacPorts.
Install FUSE for macOS from https://osxfuse.github.io/
export GOPATH="${GOPATH=$HOME}" # set GOPATH to ~ if it's not already set
go get -d github.com/rfjakob/gocryptfs
$GOPATH/src/github.com/rfjakob/gocryptfs/build-without-openssl.bash # compiles to $GOPATH/bin/gocryptfs

Initialize an encrypted directory

test -e /dev/osxfuse0 || /Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse # initialize FUSE for macOS once per boot until @bailey27's suggestion is merged in.
GOCRYPTFS_ENCRYPTED_DIR=~/secrets.encrypted
GOCRYPTFS_MOUNT_DIR=~/secrets
mkdir $GOCRYPTFS_ENCRYPTED_DIR $GOCRYPTFS_MOUNT_DIR
$GOPATH/bin/gocryptfs -init "$GOCRYPTFS_ENCRYPTED_DIR" # choose a passphrase

Unlock and mount encrypted directory and write to it

$GOPATH/bin/gocryptfs "$GOCRYPTFS_ENCRYPTED_DIR" "$GOCRYPTFS_MOUNT_DIR" # supply passphrase
echo "secret text" > $GOCRYPTFS_MOUNT_DIR/test
tree $GOCRYPTFS_MOUNT_DIR
tree $GOCRYPTFS_ENCRYPTED_DIR

Unmount and lock

umount $GOCRYPTFS_MOUNT_DIR

Unlock and remount

$GOPATH/bin/gocryptfs "$GOCRYPTFS_ENCRYPTED_DIR" "$GOCRYPTFS_MOUNT_DIR" # supply passphrase
test "$(cat $GOCRYPTFS_MOUNT_DIR/test)" == "secret text" && echo "It works!" || echo "Not working :("

It works!

Adapt tests and benchmarks for Mac

The following changes were made to get gocryptfs passing more tests and the benchmarks. No effort has yet been made to make these changes portable across operating systems so the same codebase works on Linux and Mac (and maybe the other BSDs?). My goal was to get it minimally working and run the tests to get a better handle on what's missing.

fusermount → umount

There is no fusermount with FUSE for macOS, they recommend using umount.
https://github.com/osxfuse/osxfuse/wiki/FAQ#48-how-should-i-unmount-my-fuse-for-os-x-file-system-i-cannot-find-the-fusermount-program-anywhere

perl -p -i -e 's/fusermount -u -z/umount/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/benchmark.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/benchmark-reverse.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/reverse/linux-tarball-test.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/extractloop.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/fsstress-gocryptfs.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/fsstress-loopback.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/pingpong-rsync.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/pingpong.bash

perl -p -i -e 's/fusermount -u/umount/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/test.bash

perl -p -i -e 's/"fusermount", "-u", "-z"/"umount"/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/mount.go \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/test_helpers/helpers.go

time

OS X / macOS's time does not support -f to set the format string, so let's use grep to parse out the elapsed "real" time

perl -p -i -e 's!LC_ALL=C /usr/bin/time -f %e 2>&1 $@ > /dev/null!LC_ALL=C /usr/bin/time -p 2>\&1 $@ > /dev/null | grep real | grep -Eo "[0-9.]+"!g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/benchmark-reverse.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/canonical-benchmarks.bash

stat

Linux uses stat -c %s to get size in bytes, while OS X / macOS uses stat -f %z

perl -p -i -e 's/stat -c %s/stat -f %z/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/dl-linux-tarball.bash

cat /proc/mount → mount

Instead of cat /proc/mount, parse mount, where the 3rd field is the mounted directory

perl -p -i -e 's|cat /proc/mounts|mount|'  $GOPATH/src/github.com/rfjakob/gocryptfs/test.bash
perl -p -i -e 's|cut -f2 -d" "|cut -f3 -d" "|'  $GOPATH/src/github.com/rfjakob/gocryptfs/test.bash

syscall.PathMax → 5000

Hardcode a value since there's no syscall.PathMax on Mac

perl -p -i -e 's|syscall.PathMax|5000|'  \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/test_helpers/helpers.go

Eliminate variable warnings

Declare variables to eliminate warnings

perl -p -i -e '
   s|\) \(err error\) \{|) error {|g;
   s|\(\w+ int, err error\)|(int, error)|g;
   s|\(\w+ string, err error\)|(string, error)|g;
   s|oldpath, err = dirfdAbs\(olddirfd, oldpath\)|oldpath, err := dirfdAbs(olddirfd, oldpath)|;
   s|path, err = dirfdAbs\(dirfd, path\)|path, err := dirfdAbs(dirfd, path)|g;
   undef $/; # for multiline grep
   s|\}\n\terr = syscall.Fchdir\(dirfd\)|}\n\terr := syscall.Fchdir(dirfd)|;
' $GOPATH/src/github.com/rfjakob/gocryptfs/internal/syscallcompat/sys_darwin.go

Benchmark

Benchmark on a MacBook Pro (Retina, 15-inch, Mid 2015) with 2.8 GHz Intel Core i7

$GOPATH/src/github.com/rfjakob/gocryptfs/benchmark.bash

Testing gocryptfs at /tmp/benchmark.bash.25P
WRITE: 262144000 bytes transferred in 0.787978 secs (332679266 bytes/sec)
UNTAR: 32.33
LS:    7.93
RM:    13.31

Build gocryptfs "with openssl" on OS X / mac OS

To build with OpenSSL, install it from Homebrew brew install openssl, then follow @spaghetti2514's tip and reference it in stupidgcm.go. According to @rfjakob, building with OpenSSL is not necessary since modern Macs all have AES instructions. But it seems possible to compile with OpenSSL as well for the speedup that OpenSSL offers over go's built-in crypto when your CPU doesn't have AES extensions.

perl -p -i -e 's|// +build !without_openssl|// +build !without_openssl\n// #cgo LDFLAGS: -L/usr/local/opt/openssl/lib\n// #cgo CFLAGS: -I/usr/local/opt/openssl/include|' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/internal/stupidgcm/stupidgcm.go

Run tests

$GOPATH/src/github.com/rfjakob/gocryptfs/test.bash -v

Warning: unmounting leftover filesystem: /private/tmp/gocryptfs-test-parent/591257554/v1.1-reverse.B
Warning: unmounting leftover filesystem: /private/tmp/gocryptfs-test-parent/591257554/v1.1-reverse-plaintextnames.B
gocryptfs v1.2-35-g0f40afc-dirty without_openssl; go-fuse 0ad840c; 2017-02-11 go1.7.4
github.com/jacobsa/crypto/common
github.com/hanwen/go-fuse/fuse
github.com/rfjakob/gocryptfs/internal/stupidgcm
golang.org/x/crypto/ssh/terminal
github.com/rfjakob/eme
golang.org/x/crypto/pbkdf2
github.com/rfjakob/gocryptfs/internal/syscallcompat
github.com/jacobsa/crypto/cmac
golang.org/x/crypto/scrypt
github.com/jacobsa/crypto/siv
github.com/rfjakob/gocryptfs/internal/siv_aead
github.com/rfjakob/gocryptfs/internal/tlog
github.com/rfjakob/gocryptfs/internal/ctlsock
github.com/rfjakob/gocryptfs/internal/readpassword
github.com/hanwen/go-fuse/fuse/nodefs
github.com/hanwen/go-fuse/fuse/pathfs
github.com/rfjakob/gocryptfs/internal/cryptocore
github.com/rfjakob/gocryptfs/internal/prefer_openssl
github.com/rfjakob/gocryptfs/internal/contentenc
github.com/rfjakob/gocryptfs/internal/nametransform
github.com/rfjakob/gocryptfs/internal/configfile
github.com/rfjakob/gocryptfs/internal/fusefrontend
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse
github.com/rfjakob/gocryptfs
github.com/jacobsa/crypto/common
golang.org/x/crypto/ssh/terminal
github.com/rfjakob/gocryptfs/internal/stupidgcm
github.com/jacobsa/crypto/cmac
github.com/jacobsa/crypto/siv
github.com/rfjakob/gocryptfs/internal/siv_aead
github.com/rfjakob/gocryptfs/internal/tlog
github.com/rfjakob/gocryptfs/internal/cryptocore
github.com/rfjakob/gocryptfs/internal/contentenc
github.com/rfjakob/gocryptfs/gocryptfs-xray
gocryptfs v1.2-35-g0f40afc-dirty; go-fuse 0ad840c; 2017-02-11 go1.7.4
# github.com/rfjakob/gocryptfs/tests/matrix
tests/matrix/matrix_test.go:728: st.Atim undefined (type syscall.Stat_t has no field or method Atim)
tests/matrix/matrix_test.go:728: st.Mtim undefined (type syscall.Stat_t has no field or method Mtim)
=== RUN   TestPrefixOArgs
--- PASS: TestPrefixOArgs (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs  0.010s
?     github.com/rfjakob/gocryptfs/gocryptfs-xray [no test files]
=== RUN   TestLoadV1
Unsupported on-disk format 1--- PASS: TestLoadV1 (0.00s)
=== RUN   TestLoadV2
--- PASS: TestLoadV2 (0.58s)
=== RUN   TestLoadV2PwdError
failed to unlock master key: cipher: message authentication failed
--- PASS: TestLoadV2PwdError (0.51s)
=== RUN   TestLoadV2Feature
--- PASS: TestLoadV2Feature (0.00s)
=== RUN   TestLoadV2StrangeFeature
Unsupported feature flag "StrangeFeatureFlag"--- PASS: TestLoadV2StrangeFeature (0.01s)
=== RUN   TestCreateConfFile
--- PASS: TestCreateConfFile (0.01s)
=== RUN   TestCreateConfFileAESSIV
--- PASS: TestCreateConfFileAESSIV (0.01s)
=== RUN   TestCreateConfFileRaw64
--- PASS: TestCreateConfFileRaw64 (0.01s)
=== RUN   TestIsFeatureFlagKnown
--- PASS: TestIsFeatureFlagKnown (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/configfile  1.146s
=== RUN   TestSplitRange
--- PASS: TestSplitRange (0.00s)
=== RUN   TestCiphertextRange
--- PASS: TestCiphertextRange (0.00s)
=== RUN   TestBlockNo
--- PASS: TestBlockNo (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/contentenc  0.010s
=== RUN   TestCryptoCoreNewGo15
--- PASS: TestCryptoCoreNewGo15 (0.00s)
=== RUN   TestCryptoCoreNew
--- PASS: TestCryptoCoreNew (0.00s)
=== RUN   TestNewPanic
2017/02/10 23:57:00 Unsupported key length 16
--- PASS: TestNewPanic (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/cryptocore  0.011s
=== RUN   TestSanitizePath
--- PASS: TestSanitizePath (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/ctlsock 0.009s
?     github.com/rfjakob/gocryptfs/internal/fusefrontend  [no test files]
?     github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse  [no test files]
=== RUN   TestIsLongName
--- PASS: TestIsLongName (0.00s)
=== RUN   TestPad16
--- PASS: TestPad16 (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/nametransform 0.013s
=== RUN   TestCurrentCPU
--- PASS: TestCurrentCPU (0.00s)
  prefer_test.go:8: PreferOpenSSL=false
=== RUN   TestXeonE312xx
--- PASS: TestXeonE312xx (0.00s)
=== RUN   TestPentiumG630
--- PASS: TestPentiumG630 (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/prefer_openssl  0.012s
=== RUN   TestExtpass
--- PASS: TestExtpass (0.00s)
=== RUN   TestOnceExtpass
--- PASS: TestOnceExtpass (0.00s)
=== RUN   TestTwiceExtpass
--- PASS: TestTwiceExtpass (0.00s)
=== RUN   TestExtpassEmpty
--- PASS: TestExtpassEmpty (0.01s)
=== RUN   TestStdin
--- PASS: TestStdin (0.01s)
=== RUN   TestStdinEof
--- PASS: TestStdinEof (0.01s)
=== RUN   TestStdinEmpty
--- PASS: TestStdinEmpty (0.01s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/readpassword  0.045s
=== RUN   TestKeyLens
--- PASS: TestKeyLens (0.00s)
=== RUN   TestK32
--- PASS: TestK32 (0.00s)
=== RUN   TestK64
--- PASS: TestK64 (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/siv_aead  0.010s
=== RUN   TestEncryptDecrypt
--- PASS: TestEncryptDecrypt (0.13s)
=== RUN   TestCorruption
--- PASS: TestCorruption (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/stupidgcm 0.152s
?     github.com/rfjakob/gocryptfs/internal/syscallcompat [no test files]
?     github.com/rfjakob/gocryptfs/internal/tlog  [no test files]
=== RUN   TestInit
--- PASS: TestInit (0.03s)
=== RUN   TestInitAessiv
--- PASS: TestInitAessiv (0.03s)
=== RUN   TestInitReverse
--- PASS: TestInitReverse (0.03s)
=== RUN   TestPasswd
--- PASS: TestPasswd (0.30s)
=== RUN   TestPasswdMasterkey
--- PASS: TestPasswdMasterkey (0.22s)
=== RUN   TestPasswdReverse
--- PASS: TestPasswdReverse (0.08s)
=== RUN   TestInitConfig
--- PASS: TestInitConfig (0.05s)
=== RUN   TestRo
--- PASS: TestRo (0.08s)
=== RUN   TestNonempty
--- PASS: TestNonempty (0.08s)
=== RUN   TestShadows
--- PASS: TestShadows (0.10s)
PASS
ok    github.com/rfjakob/gocryptfs/tests/cli  0.995s

Tests hang at this point. ^C^C^C has no effect. killing the ../../gocryptfs -fg -notifypid=22251 -q -wpanic -nosyslog -zerokey /tmp/gocryptfs-test-parent/827638449/default-cipher /tmp/gocryptfs-test-parent/827638449/default-plain process outputs:

=== RUN   Test1980Tar
--- PASS: Test1980Tar (0.02s)
=== RUN   TestCtlSock
--- PASS: TestCtlSock (0.11s)
=== RUN   TestOpenTruncateRead
--- PASS: TestOpenTruncateRead (0.01s)
PASS
umount(/private/tmp/gocryptfs-test-parent/827638449/default-plain): Resource busy -- try 'diskutil unmount'
exit status 1
panic: exit status 1

goroutine 1 [running]:
panic(0x41acd00, 0xc4200b2260)
  /opt/local/lib/go/src/runtime/panic.go:500 +0x1a1
github.com/rfjakob/gocryptfs/tests/test_helpers.UnmountPanic(0xc4200147c0, 0x32)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/tests/test_helpers/helpers.go:172 +0xe0
github.com/rfjakob/gocryptfs/tests/defaults.TestMain(0xc420053ee8)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/tests/defaults/main_test.go:19 +0xba
main.main()
  github.com/rfjakob/gocryptfs/tests/defaults/_test/_testmain.go:70 +0xc6
FAIL  github.com/rfjakob/gocryptfs/tests/defaults 56.836s

and then continues to hang. Killing the ../../gocryptfs -fg -notifypid=22471 -q -wpanic -nosyslog -aessiv -masterkey e7fb8f0d-2a81df9e-26611e4b-5540b218-e48aa458-c2a623af-d0c82637-1466b5f2 -openssl=true /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.B /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.C process outputs:

example_filesystems: testing with "-openssl=true"
=== RUN   TestExampleFSv04
--- PASS: TestExampleFSv04 (0.03s)
=== RUN   TestExampleFSv05
--- PASS: TestExampleFSv05 (0.03s)
=== RUN   TestExampleFSv06
--- PASS: TestExampleFSv06 (0.03s)
=== RUN   TestExampleFSv06PlaintextNames
--- PASS: TestExampleFSv06PlaintextNames (0.03s)
=== RUN   TestExampleFSv07
--- PASS: TestExampleFSv07 (0.20s)
=== RUN   TestExampleFSv07PlaintextNames
--- PASS: TestExampleFSv07PlaintextNames (0.17s)
=== RUN   TestExampleFSv09
--- PASS: TestExampleFSv09 (0.18s)
=== RUN   TestExampleFSv11
--- PASS: TestExampleFSv11 (0.13s)
=== RUN   TestExampleFSv11reverse
decryptPath: tried to decrypt "gocryptfs.longname.3b-SW_DxXiLVpverS0DDoO8RIR2D0SwoEKdqyB4zX_E=.name"!? Returning EINVAL.
-wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname.3b-SW_DxXiLVpverS0DDoO8RIR2D0SwoEKdqyB4zX_E=.name"!? Returning EINVAL.
panic: -wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname.3b-SW_DxXiLVpverS0DDoO8RIR2D0SwoEKdqyB4zX_E=.name"!? Returning EINVAL.

goroutine 60 [running]:
panic(0x41c0f60, 0xc4202320f0)
  /opt/local/lib/go/src/runtime/panic.go:500 +0x1a1
log.(*Logger).Panic(0xc420012370, 0xc420040bc0, 0x1, 0x1)
  /opt/local/lib/go/src/log/log.go:206 +0xc7
github.com/rfjakob/gocryptfs/internal/tlog.(*toggledLogger).Printf(0xc42001a6c0, 0x4214f1c, 0x34, 0xc420040c80, 0x1, 0x1)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/tlog/log.go:64 +0x245
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).rDecryptName(0xc42001e480, 0xc4202800a0, 0x44, 0xc420248280, 0x10, 0x20, 0x0, 0x0, 0x45, 0x44, ...)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:75 +0x44d
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).decryptPath(0xc42001e480, 0xc4202800a0, 0x44, 0x0, 0x0, 0x0, 0x0)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:90 +0x5bc
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).Access(0xc42001e480, 0xc4202800a0, 0x44, 0xc400000000, 0xc420216648, 0xc4202341c0)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rfs.go:237 +0x88
github.com/hanwen/go-fuse/fuse/pathfs.(*pathInode).Access(0xc4201ea090, 0x0, 0xc420216648, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/pathfs/pathfs.go:389 +0x68
github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Access(0xc42001b440, 0xc420216630, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:361 +0x65
github.com/hanwen/go-fuse/fuse.doAccess(0xc42009e620, 0xc420216480)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/opcode.go:357 +0x42
github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0xc42009e620, 0xc420216480, 0xc420078660)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:398 +0xd3
created by github.com/hanwen/go-fuse/fuse.(*Server).loop
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:375 +0x162
Cannot open config file: open /tmp/gocryptfs-test-parent/787312488/v1.1-reverse.B/gocryptfs.conf: device not configured
--- FAIL: TestExampleFSv11reverse (0.07s)
  example_filesystems_test.go:177: /tmp/gocryptfs-test-parent/787312488/v1.1-reverse.B/gocryptfs.conf missing
  helpers.go:163: mount failed: exit status 8
=== RUN   TestExampleFSv11reversePlaintextnames
decryptPath: tried to decrypt "gocryptfs.longname._9a7xcJ1f9d8WBQ3AffeY3UD7PtukytrkQpRXVOtbdU=.name"!? Returning EINVAL.
-wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname._9a7xcJ1f9d8WBQ3AffeY3UD7PtukytrkQpRXVOtbdU=.name"!? Returning EINVAL.
panic: -wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname._9a7xcJ1f9d8WBQ3AffeY3UD7PtukytrkQpRXVOtbdU=.name"!? Returning EINVAL.

goroutine 46 [running]:
panic(0x41c0f60, 0xc420142520)
  /opt/local/lib/go/src/runtime/panic.go:500 +0x1a1
log.(*Logger).Panic(0xc420012370, 0xc420132bc0, 0x1, 0x1)
  /opt/local/lib/go/src/log/log.go:206 +0xc7
github.com/rfjakob/gocryptfs/internal/tlog.(*toggledLogger).Printf(0xc42001a6c0, 0x4214f1c, 0x34, 0xc420132c80, 0x1, 0x1)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/tlog/log.go:64 +0x245
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).rDecryptName(0xc42001e280, 0xc4201402d0, 0x44, 0xc4201461e0, 0x10, 0x20, 0x0, 0x0, 0x45, 0x44, ...)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:75 +0x44d
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).decryptPath(0xc42001e280, 0xc4201402d0, 0x44, 0x0, 0x0, 0x0, 0x0)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:90 +0x5bc
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).Access(0xc42001e280, 0xc4201402d0, 0x44, 0xc400000000, 0xc420112408, 0xc42013ee70)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rfs.go:237 +0x88
github.com/hanwen/go-fuse/fuse/pathfs.(*pathInode).Access(0xc4201561b0, 0x0, 0xc420112408, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/pathfs/pathfs.go:389 +0x68
github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Access(0xc42001ae40, 0xc4201123f0, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:361 +0x65
github.com/hanwen/go-fuse/fuse.doAccess(0xc42009e540, 0xc420112240)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/opcode.go:357 +0x42
github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0xc42009e540, 0xc420112240, 0xc420106000)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:398 +0xd3
created by github.com/hanwen/go-fuse/fuse.(*Server).loop
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:375 +0x162
--- FAIL: TestExampleFSv11reversePlaintextnames (0.18s)
  example_filesystems_test.go:221: /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.B/gocryptfs.conf missing
  example_test_helpers.go:18: open /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.C/status.txt: device not configured
  example_test_helpers.go:71: open /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.C/longname_255_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: device not configured
FAIL
FAIL  github.com/rfjakob/gocryptfs/tests/example_filesystems  64.622s

@rfjakob, Can you weigh in on what you think the next steps are?

rfjakob · Answer 70 · Mon Feb 13 2017 06:16:38 GMT+0800 (China Standard Time)

@bits Great work, thanks for the detailed report!

I'll try and merge in as much of the test fixes as possible tomorrow. The hang in the tests may be caused by a hanging unmount? I'll wrap the unmount call with a timeout that prints some status info.

Not yet sure about the decryptPath: tried to decrypt "gocryptfs.longname.XXX... panic. May be a bug in gocryptfs that's hard to hit under Linux.

Paul Sarena · Answer 71 · Tue Feb 14 2017 02:50:06 GMT+0800 (China Standard Time)

According to the syslimits.h header file distributed in OS X 10.11.6, the maximum filename length of an individual file is 255 bytes, and the maximum length of a path string is 1024 bytes.

$ grep -E '\b(PATH_MAX|NAME_MAX)' /usr/include/sys/syslimits.h

#define	NAME_MAX		  255	/* max bytes in a file name */
#define	PATH_MAX		 1024	/* max bytes in pathname */

According to http://stackoverflow.com/a/32408084

From actual testing on Mac OS X Yosemite, the max path length is 1016 characters. 1017 fails.

Paul Sarena · Answer 72 · Tue Feb 14 2017 05:44:42 GMT+0800 (China Standard Time)

I forgot to mention in the previous notes that I needed to install a flock program since Mac doesn't include one. Cloning, compiling and installing the cross-platform https://github.com/discoteq/flock with cd /tmp && git clone https://github.com/discoteq/flock.git && cd flock && ./autogen.sh && make && sudo make install worked for me.

Upon closer inspection, it seems like only test.bash uses this. Perhaps flock could be avoided entirely with a shell file locking mechanism:

# Set up a race-safe lock file so that only one copy of this script will run at a time
# Use Bash's noclobber mode, which won't redirect to an existing file
SELFNAME=$(basename "$0")
TESTDIR=/tmp/gocryptfs-test-parent
LOCKFILE="$TESTDIR/$SELFNAME.lock"
if (set -o noclobber; echo "$$" > "$LOCKFILE") 2> /dev/null; then  # Lock acquired

   # Set up a trap to delete the lock file when this script exits for any reason
   trap 'rm -f "$LOCKFILE"; exit $?' EXIT

   #
   # Perform lock-protected commands
   #

   # Remove the lock file and clean up the trap for a clean exit code
   rm -f "$LOCKFILE"
   trap - EXIT
else
   echo -n "Failed to acquire lockfile: $LOCKFILE; Held by PID "
   cat "$LOCKFILE"
   exit 1
fi

rfjakob · Answer 73 · Wed Feb 15 2017 04:44:08 GMT+0800 (China Standard Time)

@bits I looked into replacing all fusermount calls with a compatability wrapper, and it turned out to add a lot of boilerplace code to all scripts. What do you think of instead placing this script in your PATH?

Paul Sarena · Answer 74 · Wed Feb 15 2017 06:38:45 GMT+0800 (China Standard Time)

@rfjakob It might work around the unmounting issue at hand, but it could have unintended consequences for users.

There might be some confusion when they apparently have a fusermount command but it fails to mount anything. Difficulty could also arise when another program checks for, and finds a fusermount, relies on it, and then inexplicably fails. Also, it's an additional step that raises the bar that many new users would have to overcome to get going with gocryptfs on a Mac, taking away from the beautiful simplicity of placing a single binary in the PATH.

Fundamentally, fusermount is a Linux-only command. macOS, FreeBSD, OpenBSD, Solaris, etc… all use umount to dismount FUSE filesystems. Perhaps it would be better to use the OS-native command rather than create a partial stub for an OS-specific one, especially since mounting/unmounting a path is a central concept with this project.

Maybe the scripts could include:

fuse-umount() { test "$(uname)" = "Linux" && fusermount -u -z $@ || umount $@; }

Replacing the fusermount -u -z calls with fuse-umount would get that aspect of the scripts running on several more platforms without too much boilerplate.

Paul Sarena · Answer 75 · Wed Feb 15 2017 08:33:18 GMT+0800 (China Standard Time)

@rfjakob Have you considered implementing gocryptfs -u MOUNTPOINT to unmount?

rfjakob · Answer 76 · Wed Feb 15 2017 16:36:31 GMT+0800 (China Standard Time)

Good points about fusermount. I'll add an internal wrapper that checks $OSTYPE to do the right thing.

About gocryptfs -u, no, I think I don't want to pull OS functionality into gocryptfs.

rfjakob · Answer 77 · Thu Feb 16 2017 06:24:34 GMT+0800 (China Standard Time)

fusermount issues should be fixed by ce2e610 .

rfjakob · Answer 78 · Fri Feb 17 2017 04:39:01 GMT+0800 (China Standard Time)

@bits I think I have adapted/fixed everything for OSX except

flock. Since flock it is available through homebrew and the bash replacement is ginormous I want to keep it as-is
build with openssl. I think if you really want it fixing the path is reasonable.
The test hang. This needs more debugging (see below)

When the tests hang again, please send a kill -ABRT to the first go process. The best way (IMO) to find the right process is to note the PID of the shell you are running the tests from (echo $$), and then, once it hangs, run pstree -ap PID to see the process tree the below the shell. At receiving the ABRT, the process should dump a backtrace to the terminal.

Example:

$ pstree -ap 3385
bash,3385
  └─test.bash,24344 ./test.bash
      └─test.bash,24348 ./test.bash
          └─go,24704 test ./...
              ├─{go},24705
              ├─{go},24706
              ├─{go},24707
              ├─{go},24708
              ├─{go},24719
              ├─{go},24724
              ├─{go},24776
              ├─{go},24777
              ├─{go},24781
              └─{go},24783
$ kill -ABRT 24704

spaghetti2514 · Answer 79 · Fri Feb 17 2017 05:20:51 GMT+0800 (China Standard Time)

@rfjakob

I installed the third party flock implementation linked by @bits, but it does not seem to support the --nonblock option that test.bash uses. So I'm running test.bash with the flock block commented out, and with the include paths added to stupidgcm's source. Everything else should be vanilla.

Here is the backtrace resulting from killing the top go process when the tests hang (with the usual test results before it omitted):

ok  	github.com/rfjakob/gocryptfs/tests/cli	2.550s
SIGABRT: abort
PC=0xd1ad3 m=0

goroutine 0 [idle]:
runtime.mach_semaphore_timedwait(0x24400000f03, 0xc43a3c80aa, 0x7fff5fbff554, 0x244, 0x3a3c80aa2001c000, 0x632ba0, 0x7fff5fbff588, 0xc8403, 0x8744ee28aa, 0xffffffff, ...)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/sys_darwin_amd64.s:428 +0x13
runtime.semasleep1(0x8744ee28aa, 0xffffffff)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/os_darwin.go:424 +0xda
runtime.semasleep.func1()
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/os_darwin.go:451 +0x33
runtime.systemstack(0x632b00)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/proc.go:1079

goroutine 40 [syscall]:
runtime.notetsleepg(0x6321d8, 0x8bb274da60, 0x1)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/lock_sema.go:257 +0x4b fp=0xc420027f40 sp=0xc420027f00
runtime.timerproc()
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/time.go:209 +0x2eb fp=0xc420027fc0 sp=0xc420027f40
runtime.goexit()
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420027fc8 sp=0xc420027fc0
created by runtime.addtimerLocked
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/time.go:116 +0xee

goroutine 1 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0xc42021bebc)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc42021beb0)
	/usr/local/Cellar/go/1.7.5/libexec/src/sync/waitgroup.go:131 +0x97
main.(*builder).do(0xc420054b60, 0xc4202fa5b0)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1330 +0x4e1
main.runTest(0x62c0a0, 0xc42000c2a0, 0x2, 0x2)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:618 +0x1328
main.main()
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/main.go:181 +0x624

goroutine 17 [syscall, 1 minutes, locked to thread]:
runtime.goexit()
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
	/usr/local/Cellar/go/1.7.5/libexec/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/Cellar/go/1.7.5/libexec/src/os/signal/signal_unix.go:28 +0x41

goroutine 41 [select, locked to thread]:
runtime.gopark(0x494e08, 0x0, 0x44b969, 0x6, 0x18, 0x2)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc420026730, 0x0, 0x18)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/select.go:423 +0x1235
runtime.selectgo(0xc420026730)
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/signal1_unix.go:304 +0x2d1
runtime.goexit()
	/usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:2086 +0x1

goroutine 8 [select]:
main.(*builder).runTest(0xc420054b60, 0xc42038c4e0, 0x3, 0xc420491e70)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1145 +0x113a
main.(*builder).do.func1(0xc42038c4e0)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1264 +0x85
main.(*builder).do.func2(0xc42021beb0, 0xc420054b60, 0xc4201dd1e0)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1321 +0x147
created by main.(*builder).do
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1327 +0x4b8

goroutine 7 [select]:
main.(*builder).runTest(0xc420054b60, 0xc4202869c0, 0x5, 0xc420495e70)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1145 +0x113a
main.(*builder).do.func1(0xc4202869c0)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1264 +0x85
main.(*builder).do.func2(0xc42021beb0, 0xc420054b60, 0xc4201dd1e0)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1321 +0x147
created by main.(*builder).do
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1327 +0x4b8

goroutine 42 [chan receive]:
main.processSignals.func1(0xc420361500)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/signal.go:21 +0x40
created by main.processSignals
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/signal.go:23 +0x93

goroutine 114 [syscall]:
syscall.Syscall(0x3, 0x4, 0xc420047b55, 0x2ab, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.read(0x4, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/syscall/zsyscall_darwin_amd64.go:973 +0x55
syscall.Read(0x4, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/syscall/syscall_unix.go:161 +0x49
os.(*File).read(0xc42012c888, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/file_unix.go:228 +0x6a
os.(*File).Read(0xc42012c888, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/file.go:101 +0x59
bytes.(*Buffer).ReadFrom(0xc4200a2d20, 0x60aba0, 0xc42012c888, 0xc420354688, 0xc420344201, 0x86866)
	/usr/local/Cellar/go/1.7.5/libexec/src/bytes/buffer.go:176 +0x155
io.copyBuffer(0x609b20, 0xc4200a2d20, 0x60aba0, 0xc42012c888, 0x0, 0x0, 0x0, 0xc420344298, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:384 +0x323
io.Copy(0x609b20, 0xc4200a2d20, 0x60aba0, 0xc42012c888, 0x3bf2a0, 0xc420344240, 0xc420354790)
	/usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:360 +0x68
os/exec.(*Cmd).writerDescriptor.func1(0x3bf2a0, 0xc420344240)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:253 +0x4d
os/exec.(*Cmd).Start.func1(0xc4202b78c0, 0xc420220880)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:370 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:371 +0x4db

goroutine 115 [chan receive]:
os/exec.(*Cmd).Wait(0xc4202b78c0, 0xc4202e6540, 0xc420355790)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:442 +0x124
main.(*builder).runTest.func1(0xc420428fc0, 0xc4202b78c0)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1142 +0x2b
created by main.(*builder).runTest
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1143 +0xc76

goroutine 134 [chan receive]:
os/exec.(*Cmd).Wait(0xc4201f0f20, 0xc420429260, 0xc420355f90)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:442 +0x124
main.(*builder).runTest.func1(0xc4203449c0, 0xc4201f0f20)
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1142 +0x2b
created by main.(*builder).runTest
	/usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1143 +0xc76

goroutine 133 [syscall]:
syscall.Syscall(0x3, 0x5, 0xc4205528e8, 0x518, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.read(0x5, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/syscall/zsyscall_darwin_amd64.go:973 +0x55
syscall.Read(0x5, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/syscall/syscall_unix.go:161 +0x49
os.(*File).read(0xc420028a90, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/file_unix.go:228 +0x6a
os.(*File).Read(0xc420028a90, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/file.go:101 +0x59
bytes.(*Buffer).ReadFrom(0xc4201d4460, 0x60aba0, 0xc420028a90, 0xc42034ee88, 0xc420429601, 0x86866)
	/usr/local/Cellar/go/1.7.5/libexec/src/bytes/buffer.go:176 +0x155
io.copyBuffer(0x609b20, 0xc4201d4460, 0x60aba0, 0xc420028a90, 0x0, 0x0, 0x0, 0xc420429678, 0x0, 0x0)
	/usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:384 +0x323
io.Copy(0x609b20, 0xc4201d4460, 0x60aba0, 0xc420028a90, 0x3bf2a0, 0xc420429620, 0xc42034ef90)
	/usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:360 +0x68
os/exec.(*Cmd).writerDescriptor.func1(0x3bf2a0, 0xc420429620)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:253 +0x4d
os/exec.(*Cmd).Start.func1(0xc4201f0f20, 0xc4202915e0)
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:370 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:371 +0x4db

rax    0xe
rbx    0x3a3c80aa
rcx    0x7fff5fbff528
rdx    0x3a3c80aa
rdi    0xf03
rsi    0x244
rbp    0x7fff5fbff560
rsp    0x7fff5fbff528
r8     0xffffffffffffffff
r9     0xc552811
r10    0x19a3b5582f0f
r11    0x202
r12    0x23933ae9deead
r13    0x239339e1f6fc8
r14    0x14a3e0cc4b14cc00
r15    0x0
rip    0xd1ad3
rflags 0x202
cs     0x7
fs     0x0
gs     0x0

Manpreet Singh · Answer 80 · Fri Feb 17 2017 08:16:59 GMT+0800 (China Standard Time)

@bits I think I have adapted/fixed everything for OSX except

@rfjakob , the fuse mount issue as in #15 (comment) is not fixed.

rfjakob · Answer 81 · Fri Feb 17 2017 17:13:27 GMT+0800 (China Standard Time)

True, it's that I'm not sure about this. I also don't call "modprobe fuse" on Linux if the kernel module is not loaded. That the path to the loader changes across osxfuse versions is also pretty scary.

spaghetti2514 · Answer 82 · Sat Feb 18 2017 04:26:23 GMT+0800 (China Standard Time)

skip openssl build

gocryptfs v1.2-59-g7fbe69b without_openssl; go-fuse 0ad840c; 2017-02-17 go1.8
# github.com/rfjakob/gocryptfs/internal/stupidgcm
internal/stupidgcm/locking.go:11:10: fatal error: 'openssl/evp.h' file not found
#include <openssl/evp.h>
         ^
1 error generated.
FAIL	github.com/rfjakob/gocryptfs [build failed]
# github.com/rfjakob/gocryptfs/internal/stupidgcm
internal/stupidgcm/locking.go:11:10: fatal error: 'openssl/evp.h' file not found
#include <openssl/evp.h>
         ^
1 error generated.

I'm not entirely sure why this is happening when the source build.bash line isn't running.

Aleksandr Glyzov · Answer 83 · Sun Feb 19 2017 02:31:35 GMT+0800 (China Standard Time)

Here is how I install gocryptfs on OSX Sierra 10.12.3 + homebrew:

CGO_CFLAGS="-I/usr/local/opt/openssl/include" CGO_LDFLAGS="-L/usr/local/opt/openssl/lib" go get -v github.com/rfjakob/gocryptfs

Of course I also did brew install openssl beforehand.
Just my 2 cents. Perhaps someone's gonna use it.

Aleksandr Glyzov · Answer 84 · Sun Feb 19 2017 02:38:47 GMT+0800 (China Standard Time)

And here is an auto-unmounting shell script with which I use it on OSX:

https://gist.github.com/aglyzov/181f079630dcb5a525497c39149e589b

Basically it's a copy of encfssh script with encfs replaced by gocryptfs.

rfjakob · Answer 85 · Mon Feb 27 2017 03:02:15 GMT+0800 (China Standard Time)

@spaghetti2514 The without_openssl build tag was not propagated to the compilation of the tests. I have added the test-without-openssl.bash script that does that. Note that you will get several

gocryptfs has been compiled without openssl support but you are still trying to use openssl

because some tests try to explicitely enable openssl, but at least it allows you to run the test suite at all.

Raffaele Sena · Answer 86 · Sun Mar 05 2017 03:17:56 GMT+0800 (China Standard Time)

@aglyzov building using "go get" gives some warning and suggests to use build.bash so a better option to build on MacOS may be:

CGO_CFLAGS="-I/usr/local/opt/openssl/include"` CGO_LDFLAGS="-L/usr/local/opt/openssl/lib" ./build.bash

or even:

export CGO_CFLAGS="-I/usr/local/opt/openssl/include" CGO_LDFLAGS="-L/usr/local/opt/openssl/lib"
./build.bash
./test.bash

Also, MacOS doesn't have flock used by test.bash but you can install:

brew install discoteq/discoteq/flock

Erland Howden · Answer 87 · Fri Apr 28 2017 11:03:09 GMT+0800 (China Standard Time)

Just wanted to make sure folks were aware I'd posted a bounty on Bountysource for this issue. It's token ($15) at this stage, because I'm not wealthy, but I'm hoping others will chip in too! Would love to see simple, stable, production-ready mac implementation. Thanks so much for all your work folks!

https://www.bountysource.com/issues/29566952-mac-os-x-support

rfjakob · Answer 88 · Sat Jul 01 2017 17:58:05 GMT+0800 (China Standard Time)

I think Mac OS X support has reached a stage where things mostly work and this generic ticket can be closed. I have updated the status at https://github.com/rfjakob/gocryptfs#platforms to state "Beta-quality" instead of "Experimental". I'd ask Mac OS X users to create separate tickets for issues you hit so we can deal with them one by one.

@erlandh thanks for the bounty, much appreciated. I'll link to the bounty in future Mac OS X bug reports.

William Entriken · Answer 89 · Tue Sep 12 2017 11:55:30 GMT+0800 (China Standard Time)

Please also update documentation at https://nuetzlich.net/gocryptfs/comparison/

Currently OSX in progress [7] can be macOS in beta

rfjakob · Answer 90 · Wed Sep 13 2017 06:09:21 GMT+0800 (China Standard Time)

Done, I've just listed MacOS after Linux

Panayotis · Answer 91 · Mon Dec 04 2017 19:51:37 GMT+0800 (China Standard Time)

Are there any documentation on how to install gocryptfs on OSX? Thank you

alexanderharm · Answer 92 · Sat Dec 16 2017 19:18:18 GMT+0800 (China Standard Time)

I created a formula for brew, see Homebrew/homebrew-core#21786.

I still have an issue with the Homebrew sandbox preventing the test to run successfully. I will comment here once they are solved and the formula is usable.

alexanderharm · Answer 93 · Sat Dec 16 2017 20:30:48 GMT+0800 (China Standard Time)

@teras: I created a Homebrew formula which can only be merged if the load_osxfuse issue is solved upstream.

In order for the installation to work you need to tick the option Allow apps downloaded from: App Store and identified developers under the Security & Privacy-tab in System Preferences!

First install OSXFuse from the website (https://osxfuse.github.io/) or via brew:

brew cask install osxfuse

Afterwards you need to reboot your machine.

Install gocryptfs from a local tap for now:

brew install alexanderharm/homebrew-gocryptfs/gocryptfs

I tested this successfully on my machine (macOS Sierra) and it also seems to run fine on @teras' machine running macOS High Sierra.

@rfjakob: maybe this can be added to the docs once it is merged into homebrew-core?

Edit: Updated installation procedure based on the feedback of @teras.

Panayotis · Answer 94 · Tue Dec 19 2017 22:34:35 GMT+0800 (China Standard Time)

@alexanderharm I got this message when I use the application:

alexanderharm · Answer 95 · Tue Dec 19 2017 22:46:00 GMT+0800 (China Standard Time)

@teras: When did you get this message? I believe this is related to the installation of osxfuse since Benjamin Fleischer is one of the authors: https://osxfuse.github.io/.

If you feel more comfortable you can also download and install osxfuse from this website and then install only gocryptfs via brew.

Panayotis · Answer 96 · Tue Dec 19 2017 23:15:26 GMT+0800 (China Standard Time)

Everything went smooth up to the time I typed
gocryptfs cipher plain
(from the demo instructions on the site).

The second time I typed them, no message was displayed, only failing:
fuse.NewServer failed: exit status 1
Maybe you should run: /Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse

Now, I tried what you said. Indeed the installer asked me to open Settings - I did so but there was no option to allow anything.
Since I am using OSX High Sierra, maybe they changed something there?

alexanderharm · Answer 97 · Tue Dec 19 2017 23:30:56 GMT+0800 (China Standard Time)

@teras: Normally it should work. Did you brew cask uninstall osxfuse before? Do you have the FUSE preference pane?

alexanderharm · Answer 98 · Tue Dec 19 2017 23:33:05 GMT+0800 (China Standard Time)

@teras: Under Security & Privacyyou need to tick the option Allow apps downloaded from: App Store and identified developers. It sounds like you are using App Store only.

Panayotis · Answer 99 · Wed Dec 20 2017 00:12:23 GMT+0800 (China Standard Time)

@alexanderharm Indeed this is the case. It seems to work now. Thank you

alexanderharm · Answer 100 · Fri Feb 02 2018 20:47:40 GMT+0800 (China Standard Time)

As of today there is a Homebrew formula for gocryptfs.

Install brew and type brew install gocryptfs.