RuntimeError: Error compiling objects for extension
likaiucas opened this issue · comments
When I follow your instruction to run "sh make.sh", I meet this problem. I have export the CUDA_HOME in ~/.bashrc
DETAIL INFO as follow:
(maskdino) root@580bd77b7030:/data/MaskDINO/maskdino/modeling/pixel_decoder/ops# sh make.sh
running build
running build_py
running build_ext
building 'MultiScaleDeformableAttention' extension
Emitting ninja build file /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] /opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/bin/nvcc -DWITH_CUDA -I/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/include -I/opt/conda/envs/maskdino/include/python3.8 -c -c /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cuda/ms_deform_attn_cuda.cu -o /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cuda/ms_deform_attn_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=MultiScaleDeformableAttention -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
FAILED: /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cuda/ms_deform_attn_cuda.o
/opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/bin/nvcc -DWITH_CUDA -I/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/include -I/opt/conda/envs/maskdino/include/python3.8 -c -c /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cuda/ms_deform_attn_cuda.cu -o /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cuda/ms_deform_attn_cuda.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=MultiScaleDeformableAttention -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
/bin/sh: 1: /opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/bin/nvcc: not found
[2/2] c++ -MMD -MF /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.o.d -pthread -B /opt/conda/envs/maskdino/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/include -I/opt/conda/envs/maskdino/include/python3.8 -c -c /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.cpp -o /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=MultiScaleDeformableAttention -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.o
c++ -MMD -MF /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.o.d -pthread -B /opt/conda/envs/maskdino/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/include -I/opt/conda/envs/maskdino/include/python3.8 -c -c /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.cpp -o /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/build/temp.linux-x86_64-cpython-38/data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=MultiScaleDeformableAttention -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /data/MaskDINO/maskdino/modeling/pixel_decoder/ops/src/cpu/ms_deform_attn_cpu.cpp:19:0:
/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
#include <cuda_runtime_api.h>
^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1666, in _run_ninja_build
subprocess.run(
File "/opt/conda/envs/maskdino/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "setup.py", line 69, in
setup(
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
self.run_command(cmd_name)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
self.build_extensions()
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 709, in build_extensions
build_ext.build_extensions(self)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
self._build_extensions_serial()
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
self.build_extension(ext)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
_build_ext.build_extension(self, ext)
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 549, in build_extension
objects = self.compiler.compile(
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 530, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1355, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/opt/conda/envs/maskdino/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1682, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
It seems like you don’t have nvcc in the folder where the build is looking: /opt/conda/pkgs/cudatoolkit-11.1.74-h6bb024c_0/bin/nvcc: not found.
Trying symlinking your nvcc to this location and see if that works.
Not this, I manually change those dependencies, anyway I solve this problem simply change a docker image...
Great to see that you have solved this problem. Issue closed.
@likaiucas would you mind sharing your MaskDINO docker?
derictly conduct "docker search" for a high stared image. keywords: pytorch 1.9