Prebuilt module for darwin-arm64 is for wrong architecture

Question

Prebuilt module for darwin-arm64 is for wrong architecture

antelle opened this issue 4 years ago · comments

Dimitri Witkowski commented 4 years ago

Prerequisites

Put an X between the brackets on this line if you have done all of the following:
- Reproduced the problem in Safe Mode: https://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
- Followed all applicable steps in the debugging guide: https://flight-manual.atom.io/hacking-atom/sections/debugging/
- Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
- Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
- Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

Hi! I discovered that the prebuilt module for Apple Silicon is actually for x64, instead of arm64.

Steps to Reproduce

curl -sSL https://github.com/atom/node-keytar/releases/download/v7.2.0/keytar-v7.2.0-electron-v85-darwin-arm64.tar.gz -O
tar -xzvf keytar-v7.2.0-electron-v85-darwin-arm64.tar.gz 
objdump -l build/Release/keytar.node 2>&1 | grep 'format.*' --only-match

Expected behavior:

format Mach-O arm64

Actual behavior:

format Mach-O 64-bit x86-64

Reproduces how often:

n/a

Versions

v7.2.0

Additional Information

n/a

Brendan Forster · Answer 1 · Fri Dec 04 2020 20:17:06 GMT+0800 (China Standard Time)

cc @dennisameling

Dennis Ameling · Answer 2 · Mon Dec 07 2020 17:08:44 GMT+0800 (China Standard Time)

@antelle hmm, that's weird. The CI log (https://travis-ci.org/github/atom/node-keytar/jobs/744665837#L770) clearly indicates Preparing to prebuild keytar@7.2.0 for electron 11.0.0 on darwin-arm64 using node-gyp. Node-gyp has supported Apple Silicon since version 7.1.0 and Keytar uses 7.1.2.

I don't have a Mac at hand currently, but will later today. Will have a closer look then. If we can't find a solution to fix the prebuild, we can simply remove the prebuild in a next release so that it'll fall back to native compilation on arm64. To be continued

$ if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then npm run prebuild-electron-arm64-mac; fi
> keytar@7.2.0 prebuild-electron-arm64-mac
> prebuild -t 11.0.0 -r electron -a arm64 --strip
prebuild info begin Prebuild version 10.0.1
prebuild info build Preparing to prebuild keytar@7.2.0 for electron 11.0.0 on darwin-arm64 using node-gyp
prebuild info find Python using Python version 3.7.5 found at "/usr/local/opt/python/bin/python3.7"
prebuild info spawn /usr/local/opt/python/bin/python3.7
prebuild info spawn args [
prebuild info spawn args   '/Users/travis/build/atom/node-keytar/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py',
prebuild info spawn args   'binding.gyp',
prebuild info spawn args   '-f',
prebuild info spawn args   'make',
prebuild info spawn args   '-I',
prebuild info spawn args   '/Users/travis/build/atom/node-keytar/build/config.gypi',
prebuild info spawn args   '-I',
prebuild info spawn args   '/Users/travis/build/atom/node-keytar/node_modules/prebuild/node_modules/node-gyp/addon.gypi',
prebuild info spawn args   '-I',
prebuild info spawn args   '/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/prebuild/electron/11.0.0/include/node/common.gypi',
prebuild info spawn args   '-Dlibrary=shared_library',
prebuild info spawn args   '-Dvisibility=default',
prebuild info spawn args   '-Dnode_root_dir=/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/prebuild/electron/11.0.0',
prebuild info spawn args   '-Dnode_gyp_dir=/Users/travis/build/atom/node-keytar/node_modules/prebuild/node_modules/node-gyp',
prebuild info spawn args   '-Dnode_lib_file=/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/prebuild/electron/11.0.0/<(target_arch)/node.lib',
prebuild info spawn args   '-Dmodule_root_dir=/Users/travis/build/atom/node-keytar',
prebuild info spawn args   '-Dnode_engine=v8',
prebuild info spawn args   '--depth=.',
prebuild info spawn args   '--no-parallel',
prebuild info spawn args   '--generator-output',
prebuild info spawn args   'build',
prebuild info spawn args   '-Goutput_dir=.'
prebuild info spawn args ]
prebuild info spawn make
prebuild info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  CXX(target) Release/obj.target/keytar/src/async.o
  CXX(target) Release/obj.target/keytar/src/main.o
  CXX(target) Release/obj.target/keytar/src/keytar_mac.o
  SOLINK_MODULE(target) Release/keytar.node
prebuild info build Stripping debug information from build/Release/keytar.node
prebuild info build Packing build/Release/keytar.node into prebuilds/keytar-v7.2.0-electron-v85-darwin-arm64.tar.gz
prebuild info build Prebuild written to prebuilds/keytar-v7.2.0-electron-v85-darwin-arm64.tar.gz
The command "if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then npm run prebuild-electron-arm64-mac; fi" exited with 0.

Dimitri Witkowski · Answer 3 · Mon Dec 07 2020 19:46:33 GMT+0800 (China Standard Time)

I don’t see arm64 being passed to node-gyp in this log, right. Rebuilding it manually for arm64 worked well for me.

Dennis Ameling · Answer 4 · Tue Dec 08 2020 04:45:37 GMT+0800 (China Standard Time)

Should be fixed in #340 - apologies for the inconvenience @antelle. Added some more details in the PR

Dimitri Witkowski · Answer 5 · Tue Dec 08 2020 05:36:29 GMT+0800 (China Standard Time)

@dennisameling interesting, you're saying node-gyp doesn't pass the architecture to the compiler correctly, however building it with electron-builder worked for me. Here's how I compiled it:

SDKROOT=macosx11.0 electron-rebuild --arch=arm64

Logs

2020-12-07T21:33:31.5540700Z gyp verb gyp gyp format was not specified; forcing "make"
2020-12-07T21:33:31.5578540Z gyp info spawn /usr/local/opt/python@3.9/bin/python3.9
2020-12-07T21:33:31.5592330Z gyp info spawn args [
2020-12-07T21:33:31.5593940Z gyp info spawn args   '/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/node-gyp/gyp/gyp_main.py',
2020-12-07T21:33:31.5595240Z gyp info spawn args   'binding.gyp',
2020-12-07T21:33:31.5596120Z gyp info spawn args   '-f',
2020-12-07T21:33:31.5596850Z gyp info spawn args   'make',
2020-12-07T21:33:31.5597560Z gyp info spawn args   '-I',
2020-12-07T21:33:31.5598820Z gyp info spawn args   '/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/build/config.gypi',
2020-12-07T21:33:31.5600030Z gyp info spawn args   '-I',
2020-12-07T21:33:31.5601280Z gyp info spawn args   '/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/node-gyp/addon.gypi',
2020-12-07T21:33:31.5602420Z gyp info spawn args   '-I',
2020-12-07T21:33:31.5603740Z gyp info spawn args   '/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/include/node/common.gypi',
2020-12-07T21:33:31.5605050Z gyp info spawn args   '-Dlibrary=shared_library',
2020-12-07T21:33:31.5605940Z gyp info spawn args   '-Dvisibility=default',
2020-12-07T21:33:31.5607250Z gyp info spawn args   '-Dnode_root_dir=/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3',
2020-12-07T21:33:31.5608940Z gyp info spawn args   '-Dnode_gyp_dir=/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/node-gyp',
2020-12-07T21:33:31.5610710Z gyp info spawn args   '-Dnode_lib_file=/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/<(target_arch)/node.lib',
2020-12-07T21:33:31.5612470Z gyp info spawn args   '-Dmodule_root_dir=/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar',
2020-12-07T21:33:31.5613730Z gyp info spawn args   '-Dnode_engine=v8',
2020-12-07T21:33:31.5614530Z gyp info spawn args   '--depth=.',
2020-12-07T21:33:31.5615320Z gyp info spawn args   '--no-parallel',
2020-12-07T21:33:31.5616200Z gyp info spawn args   '--generator-output',
2020-12-07T21:33:31.5617000Z gyp info spawn args   'build',
2020-12-07T21:33:31.5617750Z gyp info spawn args   '-Goutput_dir=.'
2020-12-07T21:33:31.5618170Z gyp info spawn args ]
2020-12-07T21:33:31.8505650Z gyp verb command build []
2020-12-07T21:33:31.8514090Z gyp verb build type Release
2020-12-07T21:33:31.8514630Z gyp verb architecture arm64
2020-12-07T21:33:31.8516330Z gyp verb node dev dir /Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3
2020-12-07T21:33:31.8532980Z gyp verb `which` succeeded for `make` /usr/bin/make
2020-12-07T21:33:31.8585700Z gyp info spawn make
2020-12-07T21:33:31.8596120Z gyp info spawn args [ 'V=1', 'BUILDTYPE=Release', '-C', 'build' ]
2020-12-07T21:33:31.8735430Z   c++ -o Release/obj.target/keytar/src/async.o ../src/async.cc '-DNODE_GYP_MODULE_NAME=keytar' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_DARWIN_USE_64_BIT_INODE=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-DV8_COMPRESS_POINTERS' '-DV8_31BIT_SMIS_ON_64BIT_ARCH' '-DV8_REVERSE_JSARGS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DOPENSSL_NO_ASM' '-DBUILDING_NODE_EXTENSION' -I/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/include/node -I/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/src -I/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/deps/openssl/config -I/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/deps/openssl/openssl/include -I/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/deps/uv/include -I/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/deps/zlib -I/Users/runner/work/keeweb-native-modules/keeweb-native-modules/node_modules/keytar/11.0.3/deps/v8/include -I../../node-addon-api  -O3 -gdwarf-2 -mmacosx-version-min=10.7 -arch arm64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++1y -stdlib=libc++ -fno-rtti -fno-strict-aliasing -MMD -MF ./Release/.deps/Release/obj.target/keytar/src/async.o.d.raw   -c

If you search arm64 in logs, you will notice that it's passed to the compiler. I haven't done anything special with it, just a regular setup on a macOS runner on GitHub Actions. The only thing I added was this environment variable: SDKROOT=macosx11.0 because it didn't want to use SDK v11 without it.

Dimitri Witkowski · Answer 6 · Tue Dec 08 2020 05:49:23 GMT+0800 (China Standard Time)

I think know why, prebuild depends on node-gyp v6, while Apple Silicon support was added in v7.1.0, as you also noted above.

Dennis Ameling · Answer 7 · Wed Dec 09 2020 01:16:40 GMT+0800 (China Standard Time)

Interesting find! I thought that it'd use the node-gyp version specified by keytar, but apparently it doesn't then. Let's just remove the prebuild altogether for now, that's the fastest/easiest way forward I think. Done in #340 👍

Dennis Ameling · Answer 8 · Wed Dec 09 2020 04:19:04 GMT+0800 (China Standard Time)

So I just tried with

node-gyp rebuild --arch=arm64 --runtime=electron --target=11.0.3 --verbose

Then the architecture is correct:

objdump -l build/Release/keytar.node 2>&1 | grep 'format.*' --only-match
format Mach-O arm64
formation for build/Release/keytar.node

So I tried to do cd node_modules/prebuild && npm i --save node-gyp@latest, then tried running prebuild again but it still created a x86_64 build. Something to investigate later, I guess.