CMake Error: could not generate ./dynet_generated_gpu-merge2.cu.o
zerovirus123 opened this issue · comments
I followed the instructions outlined in Dynet docs. These are my setups.
CMake Command
cmake .. -DEIGEN3_INCLUDE_DIR=/usr/local/include/eigen -DENABLE_CPP_EXAMPLES=ON -DBACKEND=cuda -DCUDNN_ROOT=/usr/local/cuda/lib64 -DENABLE_BOOST=ON -DPYTHON=which python
Dependencies
g++ version
g++ (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
CUDA Version
/usr/local/cuda-10.1/
CUDNN Version
7.6.5
Boost Version
Version: 1.65.1.0ubuntu1
Python Version
/home/briansia/anaconda3/bin/python
Python 3.7.3
Eigen Version
3.3.90
Running the CMake produces the following output.
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 8.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Optimization level: fast
-- Enabling Boost
-- Found Boost: /usr/include (found version "1.65.1") found components: program_options regex serialization
-- Boost dir is /usr/include
-- BACKEND: cuda
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "10.1")
CUDA_LIBRARIES: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so;/usr/local/cuda/lib64/libcurand.so
-- Found CUDNN (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Successfully include CUDNN flags
-- Eigen dir is /usr/local/include/eigen
-- Found Cython version 0.29.15
--- CUDA: CUBLAS: /usr/lib/x86_64-linux-gnu/libcublas.so RT: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so;/usr/local/cuda/lib64/libcurand.so;/usr/local/cuda/lib64/libcudnn.so
CMAKE_INSTALL_PREFIX="/usr/local"
PROJECT_SOURCE_DIR="/home/briansia/dynet"
PROJECT_BINARY_DIR="/home/briansia/dynet/build"
LIBS="/usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.65.1\;/usr/lib/x86_64-linux-gnu/libboost_regex.so.1.65.1\;/usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.65.1\;/usr/local/cuda/lib64/libcudart_static.a\;Threads::Threads\;dl\;/usr/lib/x86_64-linux-gnu/librt.so\;/usr/local/cuda/lib64/libcurand.so\;-lpthread"
EIGEN3_INCLUDE_DIR="/usr/local/include/eigen"
MKL_LINK_DIRS=""
WITH_CUDA_BACKEND="1"
CUDA_RT_FILES="libcudart_static.a\;Threads::Threads\;dl\;librt.so\;libcurand.so\;libcudnn.so"
CUDA_RT_DIRS="/usr/local/cuda/lib64\;/usr/lib/x86_64-linux-gnu"
CUDA_CUBLAS_FILES="libcublas.so"
CUDA_CUBLAS_DIRS="/usr/lib/x86_64-linux-gnu"
MSVC=""
-- Found Boost: /usr/include (found version "1.65.1") found components: system filesystem unit_test_framework
-- Configuring done
-- Generating done
-- Build files have been written to: /home/briansia/dynet/build
I could not find out why CMAKE_HAVE_LIBC_PTHREAD failed and pthread_create could not be found in the CUDA backend. However, I proceeded to run make anyways.
Here is the output from make -j 12.
The very end of the log showed that make failed because of the following lines.
dynet/CMakeFiles/dynet.dir/build.make:97: recipe for target 'dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge0.cu.o' failed
[lots of ptxas error thrown here]
Error generating file
/home/briansia/dynet/build/dynet/CMakeFiles/dynet.dir//./dynet_generated_gpu-merge2.cu.o
Does anyone have an idea as to what caused this? Thanks.
I used CMake 3.17 installed using snap. Do you think that could be the issue?
Sorry I have no clue. I tried myself today with
cmake 3.17.3
CUDA 10.2
CUDNN 8.0.1.13
and get the following errors:
/home/cyril/projects/dynet/dynet/cudnn-ops.cu(107): error: identifier "CUDNN_CONVOLUTION_FWD_PREFER_FASTEST" is undefined
/home/cyril/projects/dynet/dynet/cudnn-ops.cu(107): error: identifier "cudnnGetConvolutionForwardAlgorithm" is undefined
/home/cyril/projects/dynet/dynet/cudnn-ops.cu(190): error: identifier "CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST" is undefined
/home/cyril/projects/dynet/dynet/cudnn-ops.cu(190): error: identifier "cudnnGetConvolutionBackwardDataAlgorithm" is undefined
/home/cyril/projects/dynet/dynet/cudnn-ops.cu(209): error: identifier "CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST" is undefined
/home/cyril/projects/dynet/dynet/cudnn-ops.cu(209): error: identifier "cudnnGetConvolutionBackwardFilterAlgorithm" is undefined
6 errors detected in the compilation of "/tmp/tmpxft_00004787_00000000-6_cudnn-ops.cpp1.ii".
CMake Error at dynet_generated_cudnn-ops.cu.o.Release.cmake:276 (message):
Error generating file
/home/cyril/projects/dynet/build_cudnn_release/dynet/CMakeFiles/dynet.dir//./dynet_generated_cudnn-ops.cu.o
dynet/CMakeFiles/dynet.dir/build.make:625: recipe for target 'dynet/CMakeFiles/dynet.dir/dynet_generated_cudnn-ops.cu.o' failed
Does this look similar to yours?
I fixed my above errors. I don't know if this also fixes your errors. But with the following versions and compile flags I can compile dynet and it runs well:
System: Ubuntu 18.04 LTS
Compiler: gcc 8.4.0
cmake 3.17.3
CUDA 10.2.89-440.33.01_1.0-1
CUDNN 8.0.1.13
Eigen: 3.3.90
MKL: 2020.0.166
Cmake options (maybe you have different installation paths and GPU aschitecture):
-DCMAKE_BUILD_TYPE=Release -DRELEASE_OPT_LEVEL=3 -DCMAKE_CXX_FLAGS="-Wno-invalid-partial-specialization -Wno-c++11-narrowing " -DEIGEN3_INCLUDE_DIR=/usr/local/include/eigen3 -DMKL=TRUE -DMKL_ROOT=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl -DENABLE_BOOST=ON -DBACKEND=cuda -DCUDA_ARCH=75
With the above Eigen version you will get millions of warnings. But nevertheless it works well.