Runtime Error
VLadImirluren opened this issue · comments
I used the command as follow:
srun -p Ai4sci_3D --gres gpu:1 bash tools/dist_train.sh configs/imvoxelnet/imvoxelnet_kitti.py 1
But I meet some problem:
(imvoxel) [nanuey@SH-IDC1-10-140-24-32 imvoxelnet]$ srun -p Ai4sci_3D --gres gpu:1 bash tools/dist_train.sh configs/imvoxelnet/imvoxelnet_kitti.py 1
srun: job 2563067 queued and waiting for resources
srun: job 2563067 has been allocated resources
srun: Job 2563067 scheduled successfully!
Current QUOTA_TYPE is [reserved], which means the job has occupied quota in RESERVED_TOTAL under your partition.
Current PHX_PRIORITY is normal
Traceback (most recent call last):
File "tools/train.py", line 15, in
from mmdet3d.datasets import build_dataset
File "/mnt/petrelfs/dengken_nerf_perception/imvoxelnet/mmdet3d/datasets/init.py", line 1, in
from mmdet.datasets.builder import build_dataloader
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/datasets/init.py", line 2, in
from .cityscapes import CityscapesDataset
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/datasets/cityscapes.py", line 16, in
from .coco import CocoDataset
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/datasets/coco.py", line 14, in
from mmdet.core import eval_recalls
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/init.py", line 2, in
from .bbox import * # noqa: F401, F403
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/bbox/init.py", line 7, in
from .samplers import (BaseSampler, CombinedSampler,
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/bbox/samplers/init.py", line 9, in
from .score_hlr_sampler import ScoreHLRSampler
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/bbox/samplers/score_hlr_sampler.py", line 2, in
from mmcv.ops import nms_match
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/ops/init.py", line 1, in
from .bbox import bbox_overlaps
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/ops/bbox.py", line 3, in
ext_module = ext_loader.load_ext('_ext', ['bbox_overlaps'])
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/utils/ext_loader.py", line 11, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor6deviceEv
/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects --local_rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
FutureWarning,
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 8099) of binary: /mnt/est/nanuey/anaconda3/envs/imvoxel/bin/python3
Traceback (most recent call last):
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/run.py", line 713, in run
)(*cmd_args)
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
tools/train.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2023-11-06_19:17:25
host : SH-IDC1-10-140-24-88
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 8099)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
I have installed as your instruction, only changed the version of pytorch, I use pytorch 1.10.0, cuda 11.3, mmdet3d 0.8.0, mmdet 2.10, mmcv-full 1.2.7.
"pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
Install MMCV
pip install mmcv-full==1.2.7+torch1.6.0+cu101 -f https://openmmlab.oss-accelerate.aliyuncs.com/mmcv/dist/index.html
pip install mmdet==2.10.0
Install MMDetection
git clone https://github.com/saic-vul/imvoxelnet.git
cd imvoxelnet
pip install -r requirements/build.txt
pip install --no-cache-dir -e .
Uninstall pycocotools installed by nuscenes-devkit and reinstall mmpycocotools
pip uninstall pycocotools --no-cache-dir -y
pip install mmpycocotools==12.0.3 --no-cache-dir --force --no-deps
Install differentiable IoU
cd ..
git clone https://github.com/lilanxiao/Rotated_IoU
cp -r Rotated_IoU/cuda_op imvoxelnet/mmdet3d/ops/rotated_iou
cd imvoxelnet/mmdet3d/ops/rotated_iou/cuda_op
python setup.py install"
But I still meet the problem I show: ImportError: /mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor6deviceEv.
Could you please help me
Best wishes~
I dont know why some words so big... Ignore it please..
ImportError: /mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor6deviceEv
This basically means that your cuda version for pytorch or mmcv is wrong. E.g. you install pytorch with cuda==11.3 and mmcv with cuda==10.1.
ImportError: /mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor6deviceEv
This basically means that your
cudaversion forpytorchormmcvis wrong. E.g. you installpytorchwithcuda==11.3andmmcvwithcuda==10.1.
nonono, I was not using the mmcv installation command as you gived.
I just tried another time and using this:pip install mmcv-full==1.2.7 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html
but still the same error
ImportError: /mnt/est/nanuey/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor6deviceEv
This basically means that your
cudaversion forpytorchormmcvis wrong. E.g. you installpytorchwithcuda==11.3andmmcvwithcuda==10.1.
I mean I mainly used your commands and adjust it based on my situation: pytorch version 1.10.0 (I adjust as above)
If I understand correctly, there is no mmcv-full==1.2.7 at https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html .
If I understand correctly, there is no
mmcv-full==1.2.7at https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html .
Oh.... So....How can I solve this problem...Could you give me some advice or useful command ?.....Thanks....
You can try to download mmcv code at 1.2.7 release and build it from source with your cuda and pytorch versions.
You can try to download mmcv code at 1.2.7 release and build it from source with your cuda and pytorch versions.
I will try it.Thanks
You can try to download mmcv code at 1.2.7 release and build it from source with your cuda and pytorch versions.
I compile mmcv by" git checkout v1.2.7; pip install -v -e ." and successful. But when I run the code by the command mention above, meet a new error:
Traceback (most recent call last):
File "tools/train.py", line 15, in
from mmdet3d.datasets import build_dataset
File "/mnt/petrelfs/dengken_nerf_perception/imvoxelnet/mmdet3d/datasets/init.py", line 1, in
from mmdet.datasets.builder import build_dataloader
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/datasets/init.py", line 2, in
from .cityscapes import CityscapesDataset
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/datasets/cityscapes.py", line 16, in
from .coco import CocoDataset
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/datasets/coco.py", line 15, in
from mmdet.core import eval_recalls
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/init.py", line 2, in
from .bbox import * # noqa: F401, F403
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/bbox/init.py", line 7, in
from .samplers import (BaseSampler, CombinedSampler,
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/bbox/samplers/init.py", line 9, in
from .score_hlr_sampler import ScoreHLRSampler
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmdet/core/bbox/samplers/score_hlr_sampler.py", line 2, in
from mmcv.ops import nms_match
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/ops/init.py", line 1, in
from .bbox import bbox_overlaps
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/ops/bbox.py", line 3, in
ext_module = ext_loader.load_ext('_ext', ['bbox_overlaps'])
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/utils/ext_loader.py", line 11, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/mmcv/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops19empty_memory_format4callEN3c108ArrayRefINS2_6SymIntEEENS2_8optionalINS2_10ScalarTypeEEENS6_INS2_6LayoutEEENS6_INS2_6DeviceEEENS6_IbEENS6_INS2_12MemoryFormatEEE
/mnt/lustre/houyuenan/anaconda3/envs/imvoxel/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated
Looks like torch==1.10.0 is too good for mmcv==1.2.7 :(
Looks like torch==1.10.0 is too good for mmcv==1.2.7 :(
help me please ! I use too old version will meet the problem of 8.0 arch problem. T_T
Looks like the only solution is to use our implementation from mmdetection3d. It works with modern pytorch.
mmdetection3d
Exactly, I want to use NeRF-Det..It implement based on your excellent project...I neet to install this environment first...
Looks like the only solution is to use our implementation from mmdetection3d. It works with modern pytorch.
I use pytorch 1.8.0 but meet the problem:
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [89 lines of output]
No CUDA runtime is found, using CUDA_HOME='/mnt/cache/share/cuda-11.3/'
Compiling sparse_conv_ext without CUDA
Compiling iou3d_cuda without CUDA
Compiling voxel_layer without CUDA
Compiling roiaware_pool3d_ext without CUDA
Compiling ball_query_ext without CUDA
Compiling group_points_ext without CUDA
Compiling interpolate_ext without CUDA
Compiling furthest_point_sample_ext without CUDA
Compiling gather_points_ext without CUDA
running develop
running egg_info
writing mmdet3d.egg-info/PKG-INFO
writing dependency_links to mmdet3d.egg-info/dependency_links.txt
writing requirements to mmdet3d.egg-info/requires.txt
writing top-level names to mmdet3d.egg-info/top_level.txt
reading manifest file 'mmdet3d.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/easy_install.py:147: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
EasyInstallDeprecationWarning,
/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
setuptools.SetuptoolsDeprecationWarning,
warning: no files found matching 'mmdet3d/ops//*.cpp'
warning: no files found matching 'mmdet3d/ops//.cu'
warning: no files found matching 'mmdet3d/ops/**/.h'
warning: no files found matching 'mmdet3d/ops/**/*.cc'
warning: no files found matching 'mmdet3d/VERSION'
adding license file 'LICENSE'
writing manifest file 'mmdet3d.egg-info/SOURCES.txt'
running build_ext
building 'mmdet3d.ops.spconv.sparse_conv_ext' extension
creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build
creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37
creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d
creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d/ops
creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d/ops/spconv
creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d/ops/spconv/src
Traceback (most recent call last):
File "", line 36, in
File "", line 34, in
File "/mnt/petrelfs/dengken_nerf_perception/imvoxelnet/setup.py", line 248, in
zip_safe=False)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/develop.py", line 34, in run
self.install_for_development()
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/develop.py", line 114, in install_for_development
self.run_command('build_ext')
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
self.build_extensions()
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions
build_ext.build_extensions(self)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
self._build_extensions_serial()
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
self.build_extension(ext)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
_build_ext.build_extension(self, ext)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 556, in build_extension
depends=ext.depends,
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 524, in unix_wrap_ninja_compile
cuda_post_cflags = unix_cuda_flags(cuda_post_cflags)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 423, in unix_cuda_flags
cflags + _get_cuda_arch_flags(cflags))
File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range
[end of output]
Looks like the only solution is to use our implementation from mmdetection3d. It works with modern pytorch.
I use pytorch 1.8.0 but meet the problem:
× python setup.py develop did not run successfully. │ exit code: 1 ╰─> [89 lines of output] No CUDA runtime is found, using CUDA_HOME='/mnt/cache/share/cuda-11.3/' Compiling sparse_conv_ext without CUDA Compiling iou3d_cuda without CUDA Compiling voxel_layer without CUDA Compiling roiaware_pool3d_ext without CUDA Compiling ball_query_ext without CUDA Compiling group_points_ext without CUDA Compiling interpolate_ext without CUDA Compiling furthest_point_sample_ext without CUDA Compiling gather_points_ext without CUDA running develop running egg_info writing mmdet3d.egg-info/PKG-INFO writing dependency_links to mmdet3d.egg-info/dependency_links.txt writing requirements to mmdet3d.egg-info/requires.txt writing top-level names to mmdet3d.egg-info/top_level.txt reading manifest file 'mmdet3d.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' /mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/easy_install.py:147: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. EasyInstallDeprecationWarning, /mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. setuptools.SetuptoolsDeprecationWarning, warning: no files found matching 'mmdet3d/ops//*.cpp' warning: no files found matching 'mmdet3d/ops//.cu' warning: no files found matching 'mmdet3d/ops/**/.h' warning: no files found matching 'mmdet3d/ops/**/*.cc' warning: no files found matching 'mmdet3d/VERSION' adding license file 'LICENSE' writing manifest file 'mmdet3d.egg-info/SOURCES.txt' running build_ext building 'mmdet3d.ops.spconv.sparse_conv_ext' extension creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37 creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d/ops creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d/ops/spconv creating /mnt/petrelfs/dengken_nerf_perception/imvoxelnet/build/temp.linux-x86_64-cpython-37/mmdet3d/ops/spconv/src Traceback (most recent call last): File "", line 36, in File "", line 34, in File "/mnt/petrelfs/dengken_nerf_perception/imvoxelnet/setup.py", line 248, in zip_safe=False) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/init.py", line 87, in setup return distutils.core.setup(**attrs) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/develop.py", line 34, in run self.install_for_development() File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/develop.py", line 114, in install_for_development self.run_command('build_ext') File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run _build_ext.run(self) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run self.build_extensions() File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions build_ext.build_extensions(self) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions self._build_extensions_serial() File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial self.build_extension(ext) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 246, in build_extension _build_ext.build_extension(self, ext) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 556, in build_extension depends=ext.depends, File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 524, in unix_wrap_ninja_compile cuda_post_cflags = unix_cuda_flags(cuda_post_cflags) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 423, in unix_cuda_flags cflags + _get_cuda_arch_flags(cflags)) File "/mnt/lustre/houyuenan/anaconda3/envs/nerf_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags arch_list[-1] += '+PTX' IndexError: list index out of range [end of output]
while compiling imvoxelnet.. "pip install --no-chache-dir -e . "
Author
So bad, it seems cuda version incompatible...
Looks like the only solution is to use our implementation from mmdetection3d. It works with modern pytorch.
I use pytorch 1.8 and cuda 10.1 and compile mmcv and imvoxelnet successfully, but I meet
Traceback (most recent call last):
File "tools/train.py", line 15, in
from mmdet3d.datasets import build_dataset
File "/mnt/petrelfs/dengken_nerf_perception/imvoxelnet/mmdet3d/datasets/init.py", line 1, in
from mmdet.datasets.builder import build_dataloader
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/site-packages/mmdet/datasets/init.py", line 2, in
from .cityscapes import CityscapesDataset
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/site-packages/mmdet/datasets/cityscapes.py", line 16, in
from .coco import CocoDataset
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/site-packages/mmdet/datasets/coco.py", line 14, in
from mmdet.core import eval_recalls
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/site-packages/mmdet/core/init.py", line 2, in
from .bbox import * # noqa: F401, F403
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/site-packages/mmdet/core/bbox/init.py", line 7, in
from .samplers import (BaseSampler, CombinedSampler,
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/site-packages/mmdet/core/bbox/samplers/init.py", line 9, in
from .score_hlr_sampler import ScoreHLRSampler
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/site-packages/mmdet/core/bbox/samplers/score_hlr_sampler.py", line 2, in
from mmcv.ops import nms_match
File "/mnt/petrelfs/dengken_nerf_perception/mmcv/mmcv/ops/init.py", line 1, in
from .bbox import bbox_overlaps
File "/mnt/petrelfs/dengken_nerf_perception/mmcv/mmcv/ops/bbox.py", line 3, in
ext_module = ext_loader.load_ext('_ext', ['bbox_overlaps'])
File "/mnt/petrelfs/dengken_nerf_perception/mmcv/mmcv/utils/ext_loader.py", line 11, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/mnt/lustre/houyuenan/anaconda3/envs/nerfdet/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'mmcv._ext'
I am going to crazzzzzzy.......
If I install mmcv-full, the same error as I first mentioned occur again !!!!!
It is so unbelievable....
Looks like the only solution is to use our implementation from mmdetection3d. It works with modern pytorch.
By the way, pip install mmcv-full==1.2.7+torch1.6.0+cu101 -f https://openmmlab.oss-accelerate.aliyuncs.com/mmcv/dist/index.html, will not find a match version to install. It will error too
I success by using torch 1.8.1, cuda 10.1, mmcv1.3 mmdet 2.11. Thanks a lot