Installation issues
rkoystart opened this issue · comments
For Installing marian I followed these commands
git clone https://github.com/marian-nmt/marian
mkdir marian/build
cd marian/build
cmake ..
make -j4
and i Have given you the complete trace.
(base) rkoy@rkoy-gpu2:~/marian/build$ cmake ..
-- The CXX compiler identification is GNU 9.3.0
-- The C compiler identification is GNU 8.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Project name: marian
-- Project version: v1.10.0+6f6d4846
Submodule 'examples' (https://github.com/marian-nmt/marian-examples) registered for path 'examples'
Submodule 'regression-tests' (https://github.com/marian-nmt/marian-regression-tests) registered for path 'regression-tests'
Submodule 'src/3rd_party/fbgemm' (https://github.com/marian-nmt/FBGEMM) registered for path 'src/3rd_party/fbgemm'
Submodule 'src/3rd_party/intgemm' (https://github.com/marian-nmt/intgemm/) registered for path 'src/3rd_party/intgemm'
Submodule 'src/3rd_party/nccl' (https://github.com/marian-nmt/nccl) registered for path 'src/3rd_party/nccl'
Submodule 'src/3rd_party/sentencepiece' (https://github.com/marian-nmt/sentencepiece) registered for path 'src/3rd_party/sentencepiece'
Submodule 'src/3rd_party/simple-websocket-server' (https://github.com/marian-nmt/Simple-WebSocket-Server) registered for path 'src/3rd_party/simple-websocket-server'
Cloning into '/home/rkoy/marian/examples'...
Cloning into '/home/rkoy/marian/regression-tests'...
Cloning into '/home/rkoy/marian/src/3rd_party/fbgemm'...
Cloning into '/home/rkoy/marian/src/3rd_party/intgemm'...
Cloning into '/home/rkoy/marian/src/3rd_party/nccl'...
Cloning into '/home/rkoy/marian/src/3rd_party/sentencepiece'...
Cloning into '/home/rkoy/marian/src/3rd_party/simple-websocket-server'...
Submodule path 'examples': checked out 'c19b7814d71febf1053bd93af6ac314b46204092'
Submodule path 'regression-tests': checked out '18c4e54806205a3a29b0a8435864d6312dccaacf'
Submodule path 'src/3rd_party/fbgemm': checked out '055d2a099c829563aff1fffdeb4594ad8cfe5d99'
Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'src/3rd_party/fbgemm/third_party/asmjit'
Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'src/3rd_party/fbgemm/third_party/cpuinfo'
Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'src/3rd_party/fbgemm/third_party/googletest'
Cloning into '/home/rkoy/marian/src/3rd_party/fbgemm/third_party/asmjit'...
Cloning into '/home/rkoy/marian/src/3rd_party/fbgemm/third_party/cpuinfo'...
Cloning into '/home/rkoy/marian/src/3rd_party/fbgemm/third_party/googletest'...
Submodule path 'src/3rd_party/fbgemm/third_party/asmjit': checked out '4da474ac9aa2689e88d5e40a2f37628f302d7e3c'
Submodule path 'src/3rd_party/fbgemm/third_party/cpuinfo': checked out 'd5e37adf1406cf899d7d9ec1d317c47506ccb970'
Submodule path 'src/3rd_party/fbgemm/third_party/googletest': checked out '0fc5466dbb9e623029b1ada539717d10bd45e99e'
Submodule path 'src/3rd_party/intgemm': checked out '8abde25b13c3ab210c0dec8e23f4944e3953812d'
Submodule path 'src/3rd_party/nccl': checked out '5dcf7751494f9d04057bfc6b4a2b64611bc12253'
Submodule path 'src/3rd_party/sentencepiece': checked out '8336bbd0c1cfba02a879afe625bf1ddaf7cd93c5'
Submodule path 'src/3rd_party/simple-websocket-server': checked out '257439f5bd0a15f315c1c2733ea8a4fb0e32c1db'
CMake Warning at CMakeLists.txt:74 (message):
CMAKE_BUILD_TYPE not set; setting to Release
-- Checking support for CPU intrinsics
-- SSE2 support found [156/230]
-- SSE3 support found
-- SSE4.1 support found
-- SSE4.2 support found
-- AVX support found
-- AVX2 support found
-- AVX512 support found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr (found suitable version "10.1", minimum required is "9.0")
-- Found CUDA libraries: /usr/lib/x86_64-linux-gnu/libcurand.so /usr/lib/x86_64-linux-gnu/libcusparse.so /usr/lib/x86_64-linux-gnu/libcublas.so
-- Not Found Tcmalloc
CMake Warning at CMakeLists.txt:423 (message):
Cannot find TCMalloc library. Continuing.
-- Could NOT find MKL (missing: MKL_LIBRARIES MKL_INCLUDE_DIRS MKL_INTERFACE_LIBRARY MKL_SEQUENTIAL_LAYER_LIBRARY MKL_CORE_LIBRARY)
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Could NOT find BLAS (missing: BLAS_LIBRARIES)
-- VERSION: 0.1.94
-- Not Found TCMalloc: TCMALLOC_LIB-NOTFOUND
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Configuring done
-- Generating done
-- Build files have been written to: /home/rkoy/marian/build
(base) rkoy@rkoy-gpu2:~/marian/build$ make -j4
Scanning dependencies of target nccl_install
Scanning dependencies of target marian_version
Scanning dependencies of target sentencepiece_train-static
Scanning dependencies of target libyaml-cpp
[ 1%] Creating directories for 'nccl_install'
[ 2%] Generating ../../src/common/git_revision.h
[ 2%] Built target marian_version
[ 3%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/binary_renamed.cpp.o
Scanning dependencies of target SQLiteCpp
[ 3%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/builder.cc.o
[ 3%] Building CXX object src/3rd_party/SQLiteCpp/CMakeFiles/SQLiteCpp.dir/src/Backup.cpp.o
[ 3%] No download step for 'nccl_install'
[ 3%] No patch step for 'nccl_install'
[ 4%] No update step for 'nccl_install'
[ 5%] No configure step for 'nccl_install'
[ 5%] Performing build step for 'nccl_install'
make[3]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. [106/230]
Generating nccl.h.in > /home/rkoy/marian/build/local/include/nccl.h
Grabbing include/nccl_net.h > /home/rkoy/marian/build/local/include/nccl_net.h
Compiling init.cc > /home/rkoy/marian/build/local/obj/init.o
[ 5%] Building CXX object src/3rd_party/SQLiteCpp/CMakeFiles/SQLiteCpp.dir/src/Column.cpp.o
[ 5%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/contrib/graphbuilder.cpp.o
[ 6%] Building CXX object src/3rd_party/SQLiteCpp/CMakeFiles/SQLiteCpp.dir/src/Database.cpp.o
[ 6%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/contrib/graphbuilderadapter.cpp.o
[ 6%] Building CXX object src/3rd_party/SQLiteCpp/CMakeFiles/SQLiteCpp.dir/src/Exception.cpp.o
[ 7%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/convert.cpp.o
[ 8%] Building CXX object src/3rd_party/SQLiteCpp/CMakeFiles/SQLiteCpp.dir/src/Statement.cpp.o
[ 8%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/directives.cpp.o
[ 8%] Building CXX object src/3rd_party/SQLiteCpp/CMakeFiles/SQLiteCpp.dir/src/Transaction.cpp.o
Compiling channel.cc > /home/rkoy/marian/build/local/obj/channel.o
[ 8%] Building C object src/3rd_party/SQLiteCpp/CMakeFiles/SQLiteCpp.dir/sqlite3/sqlite3.c.o
[ 9%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/emit.cpp.o
[ 9%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/emitfromevents.cpp.o
Compiling bootstrap.cc > /home/rkoy/marian/build/local/obj/bootstrap.o
[ 10%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/emitter.cpp.o
[ 10%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/emitterstate.cpp.o
Compiling transport.cc > /home/rkoy/marian/build/local/obj/transport.o
[ 10%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/emitterutils.cpp.o
Compiling enqueue.cc > /home/rkoy/marian/build/local/obj/enqueue.o
[ 10%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/unicode_script.cc.o
Compiling group.cc > /home/rkoy/marian/build/local/obj/group.o
Compiling debug.cc > /home/rkoy/marian/build/local/obj/debug.o
Compiling proxy.cc > /home/rkoy/marian/build/local/obj/proxy.o
Compiling misc/nvmlwrap.cc > /home/rkoy/marian/build/local/obj/misc/nvmlwrap.o
[ 11%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/exceptions.cpp.o
[ 11%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/exp.cpp.o
Compiling misc/ibvwrap.cc > /home/rkoy/marian/build/local/obj/misc/ibvwrap.o
[ 12%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/memory.cpp.o
[ 12%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/node_data.cpp.o
Compiling misc/utils.cc > /home/rkoy/marian/build/local/obj/misc/utils.o
[ 12%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/nodebuilder.cpp.o
Compiling misc/argcheck.cc > /home/rkoy/marian/build/local/obj/misc/argcheck.o
[ 13%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/nodeevents.cpp.o
Compiling transport/p2p.cc > /home/rkoy/marian/build/local/obj/transport/p2p.o
[ 13%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/null.cpp.o
[ 14%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/ostream_wrapper.cpp.o
[ 14%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/parse.cpp.o
Compiling transport/shm.cc > /home/rkoy/marian/build/local/obj/transport/shm.o
[ 15%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/parser.cpp.o
Compiling transport/net.cc > /home/rkoy/marian/build/local/obj/transport/net.o
[ 15%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/regex_yaml.cpp.o
[ 15%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/scanner.cpp.o
Compiling transport/net_socket.cc > /home/rkoy/marian/build/local/obj/transport/net_socket.o
Compiling transport/net_ib.cc > /home/rkoy/marian/build/local/obj/transport/net_ib.o
[ 16%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/scanscalar.cpp.o
Compiling transport/coll_net.cc > /home/rkoy/marian/build/local/obj/transport/coll_net.o
[ 16%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/scantag.cpp.o
[ 17%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/scantoken.cpp.o
Compiling collectives/sendrecv.cc > /home/rkoy/marian/build/local/obj/collectives/sendrecv.o
[ 18%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/trainer_factory.cc.o
Compiling collectives/all_reduce.cc > /home/rkoy/marian/build/local/obj/collectives/all_reduce.o
[ 18%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/trainer_interface.cc.o
Compiling collectives/all_gather.cc > /home/rkoy/marian/build/local/obj/collectives/all_gather.o
Compiling collectives/broadcast.cc > /home/rkoy/marian/build/local/obj/collectives/broadcast.o
[ 18%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/simplekey.cpp.o
[ 18%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/singledocparser.cpp.o
Compiling collectives/reduce.cc > /home/rkoy/marian/build/local/obj/collectives/reduce.o
[ 19%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/stream.cpp.o
Compiling collectives/reduce_scatter.cc > /home/rkoy/marian/build/local/obj/collectives/reduce_scatter.o
[ 19%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/tag.cpp.o
[ 20%] Building CXX object src/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp.dir/yaml-node.cpp.o
Compiling graph/topo.cc > /home/rkoy/marian/build/local/obj/graph/topo.o
[ 20%] Built target libyaml-cpp
[ 21%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/unigram_model_trainer.cc.o
[ 21%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/word_model_trainer.cc.o
[ 21%] Built target SQLiteCpp
Scanning dependencies of target pathie-cpp
[ 21%] Building CXX object src/3rd_party/pathie-cpp/CMakeFiles/pathie-cpp.dir/src/entry_iterator.cpp.o
[ 22%] Building CXX object src/3rd_party/pathie-cpp/CMakeFiles/pathie-cpp.dir/src/errors.cpp.o
Compiling graph/paths.cc > /home/rkoy/marian/build/local/obj/graph/paths.o
[ 22%] Building CXX object src/3rd_party/pathie-cpp/CMakeFiles/pathie-cpp.dir/src/path.cpp.o
Scanning dependencies of target zlib
[ 23%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/adler32.c.o
[ 23%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/compress.c.o
[ 24%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/crc32.c.o
Compiling graph/search.cc > /home/rkoy/marian/build/local/obj/graph/search.o
[ 24%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/deflate.c.o
[ 24%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/gzclose.c.o
[ 25%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/gzlib.c.o
[ 25%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/gzread.c.o
[ 26%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/gzwrite.c.o
[ 26%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/infback.c.o
[ 27%] Building CXX object src/3rd_party/pathie-cpp/CMakeFiles/pathie-cpp.dir/src/pathie.cpp.o
[ 27%] Building CXX object src/3rd_party/pathie-cpp/CMakeFiles/pathie-cpp.dir/src/pathie_ifstream.cpp.o
[ 27%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/inffast.c.o
Compiling graph/connect.cc > /home/rkoy/marian/build/local/obj/graph/connect.o
[ 28%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/inflate.c.o
[ 28%] Building CXX object src/3rd_party/pathie-cpp/CMakeFiles/pathie-cpp.dir/src/pathie_ofstream.cpp.o
[ 29%] Building CXX object src/3rd_party/pathie-cpp/CMakeFiles/pathie-cpp.dir/src/temp.cpp.o
[ 29%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/inftrees.c.o
[ 30%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/trees.c.o
[ 30%] Built target pathie-cpp
[ 30%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/uncompr.c.o
[ 30%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/char_model_trainer.cc.o
Compiling graph/rings.cc > /home/rkoy/marian/build/local/obj/graph/rings.o
[ 31%] Building C object src/3rd_party/zlib/CMakeFiles/zlib.dir/zutil.c.o
[ 31%] Built target zlib
Scanning dependencies of target faiss
[ 32%] Building CXX object src/3rd_party/faiss/CMakeFiles/faiss.dir/Index.cpp.o
Compiling graph/trees.cc > /home/rkoy/marian/build/local/obj/graph/trees.o
Compiling graph/tuning.cc > /home/rkoy/marian/build/local/obj/graph/tuning.o
[ 32%] Building CXX object src/3rd_party/faiss/CMakeFiles/faiss.dir/IndexLSH.cpp.o
Compiling graph/xml.cc > /home/rkoy/marian/build/local/obj/graph/xml.o
[ 33%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/bpe_model_trainer.cc.o
Generating rules > /home/rkoy/marian/build/local/obj/collectives/device/Makefile.rules
In file included from /usr/include/cuda_runtime.h:83,
from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
| ^~~~~
make[5]: *** [Makefile:53: /home/rkoy/marian/build/local/obj/collectives/device/sendrecv.dep] Error 1
make[4]: *** [Makefile:50: /home/rkoy/marian/build/local/obj/collectives/device/colldevice.a] Error 2
make[3]: *** [/home/rkoy/marian/src/3rd_party/nccl/Makefile:25: src.build] Error 2
make[2]: *** [src/3rd_party/CMakeFiles/nccl_install.dir/build.make:112: src/3rd_party/nccl_install-prefix/src/nccl_install-stamp/nccl_install-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:604: src/3rd_party/CMakeFiles/nccl_install.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 34%] Building CXX object src/3rd_party/faiss/CMakeFiles/faiss.dir/VectorTransform.cpp.o
[ 34%] Building CXX object src/3rd_party/faiss/CMakeFiles/faiss.dir/utils/Heap.cpp.o
[ 35%] Building CXX object src/3rd_party/faiss/CMakeFiles/faiss.dir/utils/hamming.cpp.o
[ 35%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/sentencepiece_trainer.cc.o
[ 35%] Building CXX object src/3rd_party/faiss/CMakeFiles/faiss.dir/utils/misc.cpp.o
[ 36%] Building CXX object src/3rd_party/sentencepiece/src/CMakeFiles/sentencepiece_train-static.dir/pretokenizer_for_training.cc.o
[ 36%] Building CXX object src/3rd_party/faiss/CMakeFiles/faiss.dir/utils/random.cpp.o
[ 36%] Linking CXX static library libsentencepiece_train.a
[ 36%] Built target sentencepiece_train-static
[ 36%] Built target faiss
make: *** [Makefile:152: all] Error 2
Please let me know what has gone wrong.
Thanks in advance.
This is a CUDA version / gcc version mismatch. Upgrade your CUDA to 11.4 or downgrade your gcc to 8.
Why do you have different C++ compiler and C compiler versions?
-- The CXX compiler identification is GNU 9.3.0
-- The C compiler identification is GNU 8.4.0