RuntimeError: CUDA error: no kernel image is available for execution on the device

[AFuJianPeople]

I encountered the following error while running the "nnUNetv2_train 701 3d_fullres all -tr nnUNetTrainerUMambaBot" in your dataset.

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

GPU is P40
cuda version is 12.0 by nvidia-smi

Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand(10).to("cuda")
tensor([0.5094, 0.2796, 0.1893, 0.6431, 0.7310, 0.0044, 0.7085, 0.6361, 0.3852,
        0.5175], device='cuda:0')
>>> torch.cuda.device_count()
2
>>> torch.cuda.is_available()
True
>>> torch.version.cuda
'11.7'

Package Version Editable project location

---

acvl-utils                    0.2
asttokens                     2.4.1
attrs                         23.2.0
Automat                       22.10.0
batchgenerators               0.25
buildtools                    1.0.6
causal-conv1d                 1.1.1
certifi                       2023.11.17
charset-normalizer            3.3.2
cmake                         3.28.1
comm                          0.2.1
connected-components-3d       3.12.4
constantly                    23.10.4
contourpy                     1.2.0
cycler                        0.12.1
dicom2nifti                   2.4.9
docopt                        0.6.2
dynamic-network-architectures 0.2
einops                        0.7.0
filelock                      3.13.1
fonttools                     4.47.2
fsspec                        2023.12.2
furl                          2.1.3
future                        0.18.3
graphviz                      0.20.1
greenlet                      3.0.3
huggingface-hub               0.20.2
hyperlink                     21.0.0
idna                          3.6
imagecodecs                   2024.1.1
imageio                       2.33.1
incremental                   22.10.0
Jinja2                        3.1.3
joblib                        1.3.2
kiwisolver                    1.4.5
lazy_loader                   0.3
linecache2                    1.0.0
lit                           17.0.6
mamba-ssm                     1.1.1
MarkupSafe                    2.1.3
matplotlib                    3.8.2
MedPy                         0.4.0
monai                         1.3.0
mpmath                        1.3.0
networkx                      3.2.1
nibabel                       5.2.0
ninja                         1.11.1.1
nnunetv2                      2.1.1        /media/DataB/ykw/U-Mamba/umamba
numpy                         1.26.3
nvidia-cublas-cu11            11.10.3.66
nvidia-cuda-cupti-cu11        11.7.101
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cudnn-cu11             8.5.0.96
nvidia-cufft-cu11             10.9.0.58
nvidia-curand-cu11            10.2.10.91
nvidia-cusolver-cu11          11.4.0.1
nvidia-cusparse-cu11          11.7.4.91
nvidia-nccl-cu11              2.14.3
nvidia-nvtx-cu11              11.7.91
opencv-python                 4.9.0.80
orderedmultidict              1.0.1
packaging                     23.2
pandas                        2.1.4
pillow                        10.2.0
pip                           23.3.1
pydicom                       2.4.4
pyparsing                     3.1.1
python-dateutil               2.8.2
python-gdcm                   3.0.23
pytz                          2023.3.post1
PyYAML                        6.0.1
redo                          2.0.4
regex                         2023.12.25
requests                      2.31.0
safetensors                   0.4.1
scikit-image                  0.22.0
scikit-learn                  1.3.2
scipy                         1.11.4
seaborn                       0.13.1
setuptools                    68.2.2
SimpleITK                     2.3.1
simplejson                    3.19.2
six                           1.16.0
SQLAlchemy                    2.0.25
sympy                         1.12
threadpoolctl                 3.2.0
tifffile                      2023.12.9
tokenizers                    0.15.0
torch                         2.0.1
torchvision                   0.15.2
tqdm                          4.66.1
traceback2                    1.4.0
traitlets                     5.14.1
transformers                  4.36.2
triton                        2.0.0
Twisted                       23.10.0
typing_extensions             4.9.0
tzdata                        2023.4
unittest2                     1.1.0
urllib3                       2.1.0
wheel                         0.41.2
yacs                          0.1.8
zope.interface                6.1

[Jun]

hi @AFuJianPeople ,

I didn't have this error. could you please test to train the default nnunet? nnUNetv2_train 701 3d_fullres all

[AFuJianPeople]

Thank you for your help @JunMa11
I used default nnunet, this error was solved.
But, when I used the CUDA_VISIBLE_DEVICES=0 nnUNetv2_predict -i INPUT_FOLDER -o OUTPUT_FOLDER -f all -d 701 -c 3d_fullres -device cuda --disable_tta to inference. What is the INPUT_FOLDER? I want to use your dataset to inference. But I didn't find the test dataset.

[Jun]

Hi @AFuJianPeople ,

The testing set is also in the provided data folder imagesVal.