Build errors on M1 MacBook
roynalnaruto opened this issue · comments
I am using the commit: 3088d56d3d7787893008c3b86ac34f6621a2342c
Steps followed:
./build.sh
Error
The log for build.sh is:
go clean -cache -testcache .
rm -rf filcrypto.h filcrypto.pc libfilcrypto.a .install-filcrypto
rm -f ./runner
cd rust && cargo clean && cd ..
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
Updating crates.io index
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
Updating crates.io index
Ignored package `cbindgen v0.20.0` is already installed, use --force to override
WARN: Skipping filcrypto::convert_aggregation_inputs - (not `extern "C"`
WARN: Skip filcrypto::LOG_INIT - (not `pub`).
WARN: Skip filcrypto::LOG_INIT - (not `no_mangle`).
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
go clean -cache -testcache .
./install-filcrypto
+ auth_header=()
+ '[' -n '' ']'
++ dirname ./install-filcrypto
+ cd .
+ rust_sources_dir=rust
++ jq -r '.[].check_cpu_for_feature | select(. != null)'
+ cpu_features_required_for_optimized_release='adx
sha_ni
sha2
sse2
avx2
avx
sse4_2
sse4_1'
+ main
++ get_release_flags
++ local __features=
++ [[ ! -f /proc/cpuinfo ]]
++ echo '[get_release_flags] no /proc/cpuinfo file; falling back to Darwin feature detection'
[get_release_flags] no /proc/cpuinfo file; falling back to Darwin feature detection
+++ sysctl -a
+++ grep machdep.cpu
+++ tr '[:upper:]' '[:lower:]'
+++ grep features
++ __features=
++ feature_map=("adx:+adx" "sha_ni:+sha" "sha2:+sha2" "sse2:+sse2" "avx2:+avx2" "avx:+avx" "sse4_2:+sse4.2" "sse4_1:+sse4.1")
++ target_features=
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c adx
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sha_ni
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sha2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sse2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c avx2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c avx
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sse4_2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sse4_1
++ current_feature=0
++ '[' 1 = 0 ']'
++ echo ''
+ local __release_flags=
+ '[' 1 '!=' 1 ']'
+ echo '[install-filcrypto/main] building libfilcrypto from local sources (dir = rust)'
[install-filcrypto/main] building libfilcrypto from local sources (dir = rust)
+ build_from_source filcrypto rust ''
+ local __library_name=filcrypto
+ local __rust_sources_path=rust
+ local __release_flags=
++ git rev-parse HEAD
+ local __repo_sha1=3088d56d3d7787893008c3b86ac34f6621a2342c
+ local __repo_sha1_truncated=3088d56d3d778789
+ echo 'building from source @ 3088d56d3d778789'
building from source @ 3088d56d3d778789
++ command -v cargo
+ '[' -x /Users/narurkar/.cargo/bin/cargo ']'
++ command -v rustup
+ '[' -x /Users/narurkar/.cargo/bin/rustup ']'
+ pushd rust
~/RenProject/go-workspace/src/github.com/renproject/multichain/chain/filecoin/filecoin-ffi/rust ~/RenProject/go-workspace/src/github.com/renproject/multichain/chain/filecoin/filecoin-ffi
+ cargo --version
cargo 1.53.0-nightly (65d57e6f3 2021-04-04)
+ additional_flags=
++ uname -s
+ '[' Darwin = Darwin ']'
+ build=lipo
+ additional_flags='--targets x86_64-apple-darwin,aarch64-apple-darwin '
+ gpu_flags=,gpu
+ '[' '' == 0 ']'
+ use_multicore_sdr=multicore-sdr
+ '[' '' == 0 ']'
+ '[' '' == 1 ']'
+ '[' '' == 1 ']'
+ '[' '' == 1 ']'
+ additional_flags='--targets x86_64-apple-darwin,aarch64-apple-darwin --no-default-features --features multicore-sdr,blst,gpu'
+ echo 'Using additional build flags: --targets x86_64-apple-darwin,aarch64-apple-darwin --no-default-features --features multicore-sdr,blst,gpu'
Using additional build flags: --targets x86_64-apple-darwin,aarch64-apple-darwin --no-default-features --features multicore-sdr,blst,gpu
+ '[' -n '' ']'
++ cat rust-toolchain
+ ./scripts/build-release.sh filcrypto nightly-2021-04-24 lipo '--targets x86_64-apple-darwin,aarch64-apple-darwin --no-default-features --features multicore-sdr,blst,gpu'
+ main filcrypto nightly-2021-04-24 lipo '--targets x86_64-apple-darwin,aarch64-apple-darwin --no-default-features --features multicore-sdr,blst,gpu'
+ [[ -z filcrypto ]]
+ [[ -z nightly-2021-04-24 ]]
+ [[ -z lipo ]]
++ mktemp
+ __build_output_log_tmp=/var/folders/7h/3_b66mg534x0k6rj4dlczh340000gn/T/tmp.erNUH915
+ trap '{ rm -f $__build_output_log_tmp; }' EXIT
+ local '__rust_flags=--print native-static-libs '
+ RUSTFLAGS='--print native-static-libs '
+ cargo +nightly-2021-04-24 lipo --release --targets x86_64-apple-darwin,aarch64-apple-darwin --no-default-features --features multicore-sdr,blst,gpu
+ tee /var/folders/7h/3_b66mg534x0k6rj4dlczh340000gn/T/tmp.erNUH915
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
[INFO cargo_lipo::meta] Will build universal library for ["filcrypto"]
[INFO cargo_lipo::lipo] Building "filcrypto" for "x86_64-apple-darwin"
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
Compiling libc v0.2.99
Compiling autocfg v1.0.1
Compiling cfg-if v1.0.0
Compiling proc-macro2 v1.0.28
Compiling unicode-xid v0.2.2
Compiling syn v1.0.74
Compiling lazy_static v1.4.0
Compiling serde_derive v1.0.127
error[E0463]: can't find crate for `core`
|
= note: the `x86_64-apple-darwin` target may not be installed
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.
error: could not compile `lazy_static`
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for `core`
|
= note: the `x86_64-apple-darwin` target may not be installed
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.
error: build failed
[ERROR cargo_lipo] Failed to build "filcrypto" for "x86_64-apple-darwin": Executing "/Users/narurkar/.rustup/toolchains/nightly-2021-04-24-aarch64-apple-darwin/bin/cargo" "--color" "auto" "build" "-p" "filcrypto" "--target" "x86_64-apple-darwin" "--release" "--lib" "--no-default-features" "--features" "multicore-sdr,blst,gpu" finished with error status: exit status: 101
+ rm -f /var/folders/7h/3_b66mg534x0k6rj4dlczh340000gn/T/tmp.erNUH915
make: *** [.install-filcrypto] Error 1
I'm not on MacOS, but from the error message it seems that you don't have the x86_64-apple-darwin target installed (I also don't know if you can cross-compile on an M1). Though, could you please try to install that target via rustup target add x86_64-apple-darwin and see if that makes it work?
Thanks @vmx for the help. I had already run rustup target add x86_64-apple-darwin, but that added this target in the default toolchain which was not the specific nightly version used by filecoin-ffi (it was the stable toolchain). So I re-added the above target after having set the specific toolchain as default and at least proceeded.
But I got stuck on the following error then:
Compiling merkletree v0.21.0
error: could not find native static library `ff-derive-crypto`, perhaps an -L flag is missing?
error: aborting due to previous error
error: could not compile `fff`
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
[ERROR cargo_lipo] Failed to build "filcrypto" for "x86_64-apple-darwin": Executing "/Users/narurkar/.rustup/toolchains/nightly-2021-04-24-aarch64-apple-darwin/bin/cargo" "--color" "auto" "build" "-p" "filcrypto" "--target" "x86_64-apple-darwin" "--release" "--lib" "--no-default-features" "--features" "multicore-sdr,blst,gpu" finished with error status: exit status: 101
+ rm -f /var/folders/7h/3_b66mg534x0k6rj4dlczh340000gn/T/tmp.rPq3jXP8
make: *** [.install-filcrypto] Error 1
The commit you're using is still a WIP, so it could well be there are problems. Could you try the latest commit from #192?
Ah, ok I now used the commit: 84aedba415038e6046aebabecfd7406933c7f547, I still ended up getting those errors.
Removing the target x86_64-apple-darwin from this line at least allows me to proceed with the compilation without the above error.
When using the cgo generated files, the hwloc library cannot be found:
ld: library not found for -lhwloc
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I have already installed the hwloc formula using brew install hwloc:
$ brew info hwloc
hwloc: stable 2.5.0 (bottled), HEAD
Portable abstraction of the hierarchical topology of modern architectures
https://www.open-mpi.org/projects/hwloc/
/opt/homebrew/Cellar/hwloc/2.5.0 (957 files, 10.2MB) *
Poured from bottle on 2021-08-17 at 11:30:32
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/hwloc.rb
License: BSD-3-Clause
==> Dependencies
Build: pkg-config ✔
==> Options
--HEAD
Install HEAD version
==> Analytics
install: 16,247 (30 days), 60,002 (90 days), 261,102 (365 days)
install-on-request: 886 (30 days), 3,937 (90 days), 17,251 (365 days)
build-error: 0 (30 days)
So I tried linking it in the CGO_LDFLAGS:
$ CGO_LDFLAGS="-g -O2 -L/opt/homebrew/opt/hwloc/lib" go vet ./...
and I get this error:
# github.com/filecoin-project/filecoin-ffi/generated
Undefined symbols for architecture arm64:
"_clCreateCommandQueueWithProperties", referenced from:
opencl3::command_queue::CommandQueue::create_with_properties::he5eaa5eef3b1d3be in libfilcrypto.a(opencl3-1c7f5b0ce1bd61d8.opencl3.5sx85cca-cgu.2.rcgu.o)
"_clEnqueueSVMFree", referenced from:
opencl3::command_queue::CommandQueue::enqueue_svm_free::h043f3e0d27f6e5b9 in libfilcrypto.a(opencl3-1c7f5b0ce1bd61d8.opencl3.5sx85cca-cgu.2.rcgu.o)
"_clEnqueueSVMMemcpy", referenced from:
opencl3::command_queue::CommandQueue::enqueue_svm_mem_cpy::hcd2c2bad4506a18e in libfilcrypto.a(opencl3-1c7f5b0ce1bd61d8.opencl3.5sx85cca-cgu.2.rcgu.o)
"_clGetPipeInfo", referenced from:
cl3::memory::get_pipe_data::hab64e37ec535213c in libfilcrypto.a(cl3-f475a8013ab67ca1.cl3.2vopi5nt-cgu.2.rcgu.o)
cl3::memory::get_pipe_info::h72950da477ccac3f in libfilcrypto.a(cl3-f475a8013ab67ca1.cl3.2vopi5nt-cgu.2.rcgu.o)
"_clSetKernelArgSVMPointer", referenced from:
opencl3::kernel::Kernel::set_arg_svm_pointer::hb7bba2efcfe216d2 in libfilcrypto.a(opencl3-1c7f5b0ce1bd61d8.opencl3.5sx85cca-cgu.3.rcgu.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Try to build on the current filecoin-ffi master branch.
The build.sh script may not work if you don't have the OS X build targets, so you can install them manually if needed:
rustup target add x86_64-apple-darwin --toolchain $(cat rust-toolchain)
rustup target add aarch64-apple-darwin --toolchain $(cat rust-toolchain)
Then build ffi like:
LIBRARY_PATH=/opt/homebrew/lib LDFLAGS="-framework OpenCL" make clean
LIBRARY_PATH=/opt/homebrew/lib LDFLAGS="-framework OpenCL" make
Thank you @cryptonemo yes, the build can go through. I also need to set the CGO_LDFLAGS="-g -O2 -L/opt/homebrew/lib". Thank you :)