cblas linker error

Question

cblas linker error

hweom opened this issue 3 years ago · comments

Here's my Cargo.toml:

[dependencies]
coaster = "0.1"
juice = { version = "0.2", default-features = false } 

[features] 
default = ["native"]
native = ["juice/native"]

And then

$ cargo build

...
error: linking with `cc` failed: exit status: 1
...
  = note: /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Dot>::dot':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:272: undefined reference to `cblas_sdot'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Axpy>::axpy':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:62: undefined reference to `cblas_saxpy'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Copy>::copy':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:28: undefined reference to `cblas_scopy'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Scal>::scal':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:137: undefined reference to `cblas_sscal'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/librust_blas-370727a962638f7d.rlib(rust_blas-370727a962638f7d.rust_blas.9e9a8cb2-cgu.4.rcgu.o): in function `<f32 as rust_blas::matrix::ops::Gemm>::gemm':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/matrix/ops.rs:41: undefined reference to `cblas_sgemm'
          collect2: error: ld returned 1 exit status

$ pacman -Q | grep blas
openblas 0.3.18-1

It's interesting that juice examples compile and run just fine...

Bernhard Schuster · Answer 1 · Sat Nov 20 2021 20:46:37 GMT+0800 (China Standard Time)

Did you try setting the env variables as stated in the README?

export BLAS_LIB_DIR=/opt/blas/lib64/
export BLAS_INCLUDE_DIR=/opt/blas/include/

to wherever they end up in your case? blas is sometimes a bit finicky to get it working.

Note that this is exaserbated since blas comes with a pkg-config .pc file, whereas openblas, at least on fedora 35, does not while cblas is not available on fedora. But this is precisely the case the env vars exist.

Mikhail Balakhno · Answer 2 · Sun Nov 21 2021 05:48:37 GMT+0800 (China Standard Time)

Not sure if I'm doing it right, but here are my steps:

$ sudo pacman -Ql openblas                                                                                                                            101 ✘  10s  
openblas /usr/
openblas /usr/include/
openblas /usr/include/f77blas.h
openblas /usr/include/openblas_config.h
openblas /usr/lib/
openblas /usr/lib/cmake/
openblas /usr/lib/cmake/openblas/
openblas /usr/lib/cmake/openblas/OpenBLASConfig.cmake
openblas /usr/lib/cmake/openblas/OpenBLASConfigVersion.cmake
openblas /usr/lib/libblas.so
openblas /usr/lib/libblas.so.3
openblas /usr/lib/libopenblas.so
openblas /usr/lib/libopenblas.so.3
openblas /usr/lib/libopenblasp-r0.3.18.so
openblas /usr/lib/pkgconfig/
openblas /usr/lib/pkgconfig/blas.pc
openblas /usr/lib/pkgconfig/openblas.pc
openblas /usr/share/
openblas /usr/share/licenses/
openblas /usr/share/licenses/openblas/
openblas /usr/share/licenses/openblas/LICENSE

export BLAS_LIB_DIR=/usr/lib/
export BLAS_INCLUDE_DIR=/usr/include/
export BLAS_VARIANT=openblas

cargo clean && cargo build

...
  = note: /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Dot>::dot':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:272: undefined reference to `cblas_sdot'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Axpy>::axpy':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:62: undefined reference to `cblas_saxpy'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Copy>::copy':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:28: undefined reference to `cblas_scopy'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/libcoaster_blas-450fb0dd29f3d07a.rlib(coaster_blas-450fb0dd29f3d07a.coaster_blas.72d24db0-cgu.4.rcgu.o): in function `<f32 as rust_blas::vector::ops::Scal>::scal':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/vector/ops.rs:137: undefined reference to `cblas_sscal'
          /usr/bin/ld: /home/mikhail/Documents/evolution/target/debug/deps/librust_blas-370727a962638f7d.rlib(rust_blas-370727a962638f7d.rust_blas.9e9a8cb2-cgu.4.rcgu.o): in function `<f32 as rust_blas::matrix::ops::Gemm>::gemm':
          /home/mikhail/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-blas-0.1.3/src/matrix/ops.rs:41: undefined reference to `cblas_sgemm'
          collect2: error: ld returned 1 exit status

Bernhard Schuster · Answer 3 · Sun Nov 21 2021 05:53:09 GMT+0800 (China Standard Time)

Thanks for the info, I'll take a look into what's going on here

Mikhail Balakhno · Answer 4 · Mon Nov 22 2021 04:15:39 GMT+0800 (China Standard Time)

Note that it easily reproduces with

git clone https://github.com/spearow/juice.git
cd juice
cargo build --release --no-default-features --features "native"

Bernhard Schuster · Answer 5 · Tue Nov 23 2021 17:32:15 GMT+0800 (China Standard Time)

Since I am not a ArchLinux user anymore, could you provide me a container (Containerfile, Dockerfile or just a link to a registry) so I can replicate your installed libs?

Bernhard Schuster · Answer 6 · Tue Nov 23 2021 17:45:30 GMT+0800 (China Standard Time)

If I install the package openblas which removes cblas, and use your provided env flags

export BLAS_LIB_DIR=/usr/lib/
export BLAS_INCLUDE_DIR=/usr/include/
export BLAS_VARIANT=openblas

it finds openblas and links juice just fine.

cargo build --release --no-default-features --features "native" -p juice
// snip
   Compiling juice v0.3.0 (/x/juice)
warning: `coaster-blas` (lib) generated 56 warnings (49 duplicates)
    Finished release [optimized] target(s) in 4.38s

I realized that adding the examples to the repo gives some false positives when compiling the examples, which require cuda right now:

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.0.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.1.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.10.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.11.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.12.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.13.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.14.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.15.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.2.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.3.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.4.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.5.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.6.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.7.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.8.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.example_rnn_regression.f81d7c26-cgu.9.rcgu.o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a.3trdu4z1kglma9yz.rcgu.o" "-Wl,--as-needed" "-L" "/x/target/release/deps" "-L" "/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/x/target/release/deps/libdocopt-524d597100669d56.rlib" "/x/target/release/deps/libstrsim-47c1da387c04b25a.rlib" "/x/target/release/deps/libenv_logger-69f1b107b026a0d3.rlib" "/x/target/release/deps/libatty-2f6770fc84208eed.rlib" "/x/target/release/deps/libtermcolor-47612503f587db35.rlib" "/x/target/release/deps/libhumantime-29d8c0b2c3ef4a0c.rlib" "/x/target/release/deps/libregex-b95032f6d10cf669.rlib" "/x/target/release/deps/libregex_syntax-99c32f5c08941720.rlib" "/x/target/release/deps/libaho_corasick-f322f133a91a8ecd.rlib" "/x/target/release/deps/libcsv-164bfdf4170c5e44.rlib" "/x/target/release/deps/libryu-b180b1eb79d044ee.rlib" "/x/target/release/deps/libitoa-6175d34776e7b699.rlib" "/x/target/release/deps/libcsv_core-10019def646a9a42.rlib" "/x/target/release/deps/libbstr-ddd3641b469990e1.rlib" "/x/target/release/deps/libregex_automata-94244832ff84c902.rlib" "/x/target/release/deps/libmemchr-34078201487ae117.rlib" "/x/target/release/deps/libjuice-e8fe7b68997d10e0.rlib" "/x/target/release/deps/libcoaster_blas-1d5d028c66910128.rlib" "/x/target/release/deps/librust_blas-166973c7e8652f79.rlib" "/x/target/release/deps/libcapnp-e467e2139bf13fad.rlib" "/x/target/release/deps/libtimeit-5822333272b6ed1e.rlib" "/x/target/release/deps/libtime-4151a6a76d87566d.rlib" "/x/target/release/deps/libserde-603334ed46f247ca.rlib" "/x/target/release/deps/libfs_err-30d1a90a80deead0.rlib" "/x/target/release/deps/libcoaster_nn-26508d680ae518f6.rlib" "/x/target/release/deps/librand-ec1d8659ab00854c.rlib" "/x/target/release/deps/librand_chacha-edbda9611d86292b.rlib" "/x/target/release/deps/libppv_lite86-0bf702597964910b.rlib" "/x/target/release/deps/librand_core-300d3c81fb3dd19a.rlib" "/x/target/release/deps/libgetrandom-30072aa4404bbea2.rlib" "/x/target/release/deps/libcoaster-67218e213fd75c38.rlib" "/x/target/release/deps/librcudnn-d971d7b38e2f5386.rlib" "/x/target/release/deps/librcudnn_sys-2fd6ab6660c2f79e.rlib" "/x/target/release/deps/librcublas-4d89aad957eed906.rlib" "/x/target/release/deps/libthiserror-0a67d38bf3445322.rlib" "/x/target/release/deps/liblog-6c337023950b012a.rlib" "/x/target/release/deps/libcfg_if-c064eaa7abc3d650.rlib" "/x/target/release/deps/librcublas_sys-83c773c08e6051d0.rlib" "/x/target/release/deps/libnum-40ea4ebf302bd9fd.rlib" "/x/target/release/deps/libnum_iter-49661d8d390052e1.rlib" "/x/target/release/deps/libnum_rational-a4afbd387cdf05a8.rlib" "/x/target/release/deps/libnum_complex-1682f0009ea2a353.rlib" "/x/target/release/deps/libnum_bigint-af3970a06d31f990.rlib" "/x/target/release/deps/libnum_integer-e2f2baa18f7b5bf4.rlib" "/x/target/release/deps/libbyteorder-1dcf00b05ddcdb79.rlib" "/x/target/release/deps/liblibc-1830b8a42fb39a31.rlib" "/x/target/release/deps/liblazy_static-ca9d39f825b354e1.rlib" "/x/target/release/deps/libenum_primitive-6950ab977d3cf50f.rlib" "/x/target/release/deps/libnum_traits-03123bd2e5ff531b.rlib" "/x/target/release/deps/libnum_traits-82d981c5cdb489a5.rlib" "/x/target/release/deps/libbitflags-5eebfebe6ed85156.rlib" "-Wl,--start-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-1165f94ac1ecb87d.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-a94c82b089904326.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-08da89f6801d2906.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-efd61e674d16f74c.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-8acd4ced312e0e4c.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-b99e1c847bd7b013.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-21f55880014c5032.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-56352f011c73d80e.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-59ef9ea6a9717497.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-0435a75812955b79.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-558037bf295e4d0d.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-6a34e7f699451c36.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-055e710ab7ace102.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-e15536f14cf1b2c5.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-0a4f11dfe00e3e85.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-273ca6e90ef8d9cf.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-ef5d8cbac66e0318.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-93a72d5d18e29f4f.rlib" "-Wl,--end-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-0c7e43ac4e512046.rlib" "-Wl,-Bdynamic" "-lopenblas" "-lcuda" "-lcudnn" "-lcudart" "-lcuda" "-lcublas" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/x/target/release/deps/example_rnn_regression-652e0b9b599c001a" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: /usr/sbin/ld: cannot find -lopenblas
          /usr/sbin/ld: cannot find -lcuda
          /usr/sbin/ld: cannot find -lcudnn
          /usr/sbin/ld: cannot find -lcudart
          /usr/sbin/ld: cannot find -lcuda
          /usr/sbin/ld: cannot find -lcublas
          collect2: error: ld returned 1 exit status
          

warning: `example-rnn-regression` (bin "example-rnn-regression") generated 17 warnings
error: could not compile `example-rnn-regression` due to previous error; 17 warnings emitted
warning: build failed, waiting for other jobs to finish...
error: build failed
[root@fa46cfedb79d x]# cargo build --release --no-default-features --features "native"

Mikhail Balakhno · Answer 7 · Mon Nov 29 2021 09:56:21 GMT+0800 (China Standard Time)

it finds openblas and links juice just fine.

juice is a library, so it wouldn't complain about missing symbols, I think?

I created a Docker image that reproduces the issue:

docker run -it hweom/juice-test
# inside the container
source /root/.profile
cd /opt/juice-test/
cargo build --release

Bernhard Schuster · Answer 8 · Tue Nov 30 2021 00:39:57 GMT+0800 (China Standard Time)

I see.

This is an issue with the provided openblas lib provided in archlinux - it doesn't seem to contain cblas_saxpy which we need. No idea why.

You are correct, that is only fails if linking with dynamic linkage, I forgot about BLAS_STATIC=1 which I use as much as possible to avoid any breakage like this.

That said, I am not entirely sure how to fix this for you, this either needs a patch to ./rust-blas contained in the repo, or the archlinux package is busted, upstream contains the required fn https://github.com/xianyi/OpenBLAS/blob/develop/cblas.h#L111

Mikhail Balakhno · Answer 9 · Tue Nov 30 2021 13:10:48 GMT+0800 (China Standard Time)

This is probably because it's compiled without CBLAS support: https://github.com/archlinux/svntogit-community/blob/d3a8b289814a2ff9a1974b447a4e8234f32c63df/trunk/PKGBUILD#L23, although I have no idea why.

I'll ask the maintainer, thanks!

Mikhail Balakhno · Answer 10 · Tue Nov 30 2021 13:50:08 GMT+0800 (China Standard Time)

Archlinux openblas package seems to be abandoned. I resolved the issue by switching to AUR's openblas-lapack. Now it builds without issues.

Thanks for the help!

Bernhard Schuster · Answer 11 · Tue Nov 30 2021 17:29:45 GMT+0800 (China Standard Time)

Could you create a PR with note so other people don't run into the same issue? That'd be much appreciated!

Mikhail Balakhno · Answer 12 · Thu Dec 02 2021 10:22:16 GMT+0800 (China Standard Time)

You mean mention this in the Juice README?