Failed to install gotch on MacOS

Question

Failed to install gotch on MacOS

HaoboGu opened this issue 3 years ago · comments

I encountered problems when I was trying to install gotch on MacOS

I've installed libtorch at /usr/local/lib/libtorch and executed setup-gotch.sh after setting environment variables. Here is the result:

GOPATH:'/usr/local/Cellar/go/1.15.6'
GOTCH_VERSION: 'v0.3.8'
CUDA_VERSION: '10.1'
go: creating new go.mod: module github.com/sugarme/gotch-test
/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.8/libtch/dummy_cuda_dependency.cpp existing. Deleting...
/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.8/libtch/lib.go existing. Deleting...
creating /usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.8/libtch/dummy_cuda_dependency.cpp for GPU
creating /usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.8/libtch/lib.go for GPU
sudo: ldconfig: command not found

however, I cannot use gotch in my go project.
I also tried to install gotch using go get, but I got

$ go get github.com/sugarme/gotch
go: github.com/sugarme/gotch upgrade => v0.3.11
# github.com/sugarme/gotch/libtch
ld: warning: directory not found for option '-L/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.11/libtch/libtorch/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib64'
ld: library not found for -lcuda
clang: error: linker command failed with exit code 1 (use -v to see invocation)

sugarme · Answer 1 · Wed Jun 30 2021 09:58:43 GMT+0800 (China Standard Time)

@HaoboGu,

As you install for your Mac OS, I don't think you can install with CUDA support, however you can try CPU version of libtorch. The installation guide is just for Linux and there might be some changes for MacOS. We have limited experience with Mac OS and gotch requires C++17 support, so make sure your xcode version support C++ 17. However you can try it and let us know how you go: https://github.com/sugarme/gotch#cpu

Please ignore ldconfig warning/error as it is not applicable for MacOS. Also make sure you have GOPATH set in your machine that can be acccessible for current user without sudo.

I hope I can find my time to update my box MacOS and update the installation but please give further feedback so that we can update installation for MacOS. Thanks.

HaoboGu · Answer 2 · Wed Jun 30 2021 10:26:46 GMT+0800 (China Standard Time)

@sugarme I've managed to install gotch by switching to cpu version. But I got another error when I tried to ran the example:

torch_api.cpp:1:9: fatal error: 'torch/csrc/autograd/engine.h' file not found

I checked the libtorch folder, the file is there.

$ ls /usr/local/lib/libtorch/include/torch/csrc/autograd
FunctionsManual.h          cpp_hook.h                 function.h                 grad_mode.h                python_anomaly_mode.h      python_fft_functions.h     python_linalg_functions.h  record_function_ops.h      variable.h
VariableTypeUtils.h        custom_function.h          function_hook.h            input_buffer.h             python_autograd.h          python_function.h          python_nn_functions.h      saved_variable.h
anomaly_mode.h             edge.h                     functions                  input_metadata.h           python_cpp_function.h      python_hook.h              python_variable.h          symbolic.h
autograd.h                 engine.h                   generated                  profiler.h                 python_engine.h            python_legacy_variable.h   python_variable_indexing.h utils

sugarme · Answer 3 · Wed Jun 30 2021 11:15:53 GMT+0800 (China Standard Time)

@HaoboGu,

Such error means your libtorch is not in the PATH. Can you printenv and check? You can set your environement for current session by doing:

export GOTCH_LIBTORCH="/usr/local/lib/libtorch"
export LIBRARY_PATH="$LIBRARY_PATH:$GOTCH_LIBTORCH/lib"
export CPATH="$CPATH:$GOTCH_LIBTORCH/lib:$GOTCH_LIBTORCH/include:$GOTCH_LIBTORCH/include/torch/csrc/api/include"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GOTCH_LIBTORCH/lib"

HaoboGu · Answer 4 · Wed Jun 30 2021 11:26:24 GMT+0800 (China Standard Time)

@sugarme ,

I added those environment variables and another error occurs

go build main.go
# github.com/sugarme/gotch/libtch
ld: warning: directory not found for option '-L/lib64'
ld: warning: directory not found for option '-L/libtorch/lib'
ld: warning: ignoring file /usr/local/lib/libtorch/lib/libtorch.so, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 )
ld: warning: ignoring file /usr/local/lib/libtorch/lib/libc10.so, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 )
ld: warning: ignoring file /usr/local/lib/libtorch/lib/libtorch_cpu.so, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 )
Undefined symbols for architecture x86_64:
  "at::__lshift__(at::Tensor const&, c10::Scalar)", referenced from:
      _atg___lshift__ in _x010.o
  "at::__lshift__(at::Tensor const&, at::Tensor const&)", referenced from:
...
...
...
...
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for torch::optim::RMSprop", referenced from:
      torch::optim::RMSprop::RMSprop(std::__1::vector<torch::optim::OptimizerParamGroup, std::__1::allocator<torch::optim::OptimizerParamGroup> >, torch::optim::RMSpropOptions) in _x010.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for torch::optim::Optimizer", referenced from:
      torch::optim::Optimizer::Optimizer(std::__1::vector<torch::optim::OptimizerParamGroup, std::__1::allocator<torch::optim::OptimizerParamGroup> >, std::__1::unique_ptr<torch::optim::OptimizerOptions, std::__1::default_delete<torch::optim::OptimizerOptions> >) in _x010.o
      torch::optim::Optimizer::~Optimizer() in _x010.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

sugarme · Answer 5 · Wed Jun 30 2021 11:43:51 GMT+0800 (China Standard Time)

@HaoboGu ,

can you just try go run . instead of go build? and try to setup a simple example to test that gotch is working in your system first.
Something like:

mkdir test && cd test
go mod init github.com/SOME_USER/test
go get github.com/sugarme/gotch@v0.3.10
touch main.go

package main
import (
	"fmt"

	"github.com/sugarme/gotch"
	ts "github.com/sugarme/gotch/tensor"
)

func main() {

xs := ts.MustRand([]int64{3, 5, 6}, gotch.Float, gotch.CPU)
fmt.Printf("%8.3f\n", xs)
fmt.Printf("%i", xs)

}

go run .

HaoboGu · Answer 6 · Wed Jun 30 2021 15:40:59 GMT+0800 (China Standard Time)

@sugarme

I got the same error when I ran go run .

sugarme · Answer 7 · Wed Jun 30 2021 16:29:32 GMT+0800 (China Standard Time)

@HaoboGu,

Do you happen to have Pytorch Python installed in your system?

HaoboGu · Answer 8 · Thu Jul 01 2021 11:13:30 GMT+0800 (China Standard Time)

@HaoboGu,

Do you happen to have Pytorch Python installed in your system?

@sugarme

Sure, I installed pytorch 1.7.0

sugarme · Answer 9 · Thu Jul 01 2021 11:45:31 GMT+0800 (China Standard Time)

@HaoboGu,

I had library linking incorrectly but the other way around, occuring with Python instead of Go. Anyway, I realize that there's DYLD_LIBRARY_PATH in MacOS instead of LD_LIBRARY_PATH so can you try to export:

export DYLD_LIBRARY_PATH="$DYLD_LIBRARY_PATH:$GOTCH_LIBTORCH/lib"

HaoboGu · Answer 10 · Thu Jul 01 2021 23:28:27 GMT+0800 (China Standard Time)

@sugarme ,

I fixed previous problems by reinstall libtorch mac cpu version, but when I ran go run . or go build, another error occured:

/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.10/libtch/tensor.go:100:20: cannot use c_ptr (type *_Ctype_long) as type *_Ctype_longlong in assignment
/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.10/libtch/tensor.go:142:54: cannot use cindexes (type *_Ctype_long) as type *_Ctype_longlong in assignment
/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.10/libtch/tensor.go:151:53: cannot use cindexes (type *_Ctype_long) as type *_Ctype_longlong in assignment

sugarme · Answer 11 · Fri Jul 02 2021 05:19:16 GMT+0800 (China Standard Time)

@HaoboGu,

Those errors actually bugs in library although it compiled Okay in Linux. Please do change the following in your machine to see how you go. I will fix and release new version if running well.

Change the following lines in /usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.10/libtch/tensor.go

// line 99: c_ptr := (*C.long)(ptr)
c_ptr := (*C.longlong)(ptr)

// line 140: cindexes := (*C.long)(indexes)
cindexes := (*C.longlong)(indexes)

// line 149: cindexes := (*C.long)(indexes)
cindexes := (*C.longlong)(indexes)

Please let me know whether this works? Thanks

HaoboGu · Answer 12 · Fri Jul 02 2021 10:56:11 GMT+0800 (China Standard Time)

@sugarme

It works! Thank you for your kind and patient reply!
I'm closing this.