ImportError: No module named 'upfirdn2d_plugin'
harisreedhar opened this issue · comments
I got this warning...
(stitenv) hari@hari-MS-7C02:/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT$ python train.py --input_folder ./data/obama --output_folder ./training_results/obama --run_name obama --num_pti_steps 80
Number of images: 200
Aligning images
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:01<00:00, 117.42it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:04<00:00, 47.81it/s]
Aligning completed
Loading e4e over the pSp framework from checkpoint: ./pretrained_models/e4e_ffhq_encode.pt
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
Loading model from: /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/lpips/weights/v0.1/alex.pth
Calculating initial inversions
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:37<00:00, 5.30it/s]
Fine tuning generator
0%| | 0/80 [00:00<?, ?it/s]Setting up PyTorch plugin "bias_act_plugin"... Failed!
/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/bias_act.py:50: UserWarning: Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:
Traceback (most recent call last):
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/bias_act.py", line 48, in _init
_plugin = custom_ops.get_plugin('bias_act_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
File "/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/custom_ops.py", line 110, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1302, in _jit_compile
is_standalone=is_standalone)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1407, in _write_ninja_file_and_build_library
error_prefix=f"Error building extension '{name}'")
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'bias_act_plugin': [1/2] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output bias_act.cuda.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/TH -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/THC -isystem /home/hari/anaconda3/envs/stitenv/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/bias_act.cu -o bias_act.cuda.o
FAILED: bias_act.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output bias_act.cuda.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/TH -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/THC -isystem /home/hari/anaconda3/envs/stitenv/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/bias_act.cu -o bias_act.cuda.o
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
ninja: build stopped: subcommand failed.
warnings.warn('Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())
Setting up PyTorch plugin "upfirdn2d_plugin"... Failed!
/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/upfirdn2d.py:34: UserWarning: Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:
Traceback (most recent call last):
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/upfirdn2d.py", line 32, in _init
_plugin = custom_ops.get_plugin('upfirdn2d_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
File "/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/custom_ops.py", line 110, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1302, in _jit_compile
is_standalone=is_standalone)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1407, in _write_ninja_file_and_build_library
error_prefix=f"Error building extension '{name}'")
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'upfirdn2d_plugin': [1/2] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d.cuda.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/TH -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/THC -isystem /home/hari/anaconda3/envs/stitenv/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/upfirdn2d.cu -o upfirdn2d.cuda.o
FAILED: upfirdn2d.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d.cuda.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/TH -isystem /home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/include/THC -isystem /home/hari/anaconda3/envs/stitenv/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/upfirdn2d.cu -o upfirdn2d.cuda.o
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
ninja: build stopped: subcommand failed.
warnings.warn('Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())
Setting up PyTorch plugin "upfirdn2d_plugin"... Failed!
/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/upfirdn2d.py:34: UserWarning: Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:
Traceback (most recent call last):
File "/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/ops/upfirdn2d.py", line 32, in _init
_plugin = custom_ops.get_plugin('upfirdn2d_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
File "/mnt/95d2aa3d-99e9-4600-91b1-2fcecff0dec5/AI_Tools/STIT/torch_utils/custom_ops.py", line 110, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1317, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1699, in _import_module_from_library
file, path, description = imp.find_module(module_name, [path])
File "/home/hari/anaconda3/envs/stitenv/lib/python3.7/imp.py", line 296, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'upfirdn2d_plugin'
What version of cuda do you have?
Can you please give the output of nvcc --version?
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
@rotemtzaban same issue, this is my output
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:35_Pacific_Daylight_Time_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.relgpu_drvr445TC445_37.28845127_0
StyleGAN2 requires having CUDA toolkit 11.0 or later. I will update the requirements to reflect this.
In the meantime you can look at the requirements here: https://github.com/NVlabs/stylegan2-ada-pytorch
@sam598
What gpu are you running on? If it's an RTX3090 you should use CUDA 11.1 or later.
If not, are you sure it's the same issue? can you send your error message?
@rotemtzaban I updated pytorch version 1.8 to 1.10 (cuda 11.1) and it worked.
Thanks for your great work :)
@rotemtzaban I have an RTX 3090 and just upgraded from CUDA 11.0 to 11.3. I am getting the same error with both versions of CUDA.
C:\Users\...\STIT>python train.py --input_folder .\datasets\obama --output_folder .\training_results\obama --run_name obama --num_pti_steps 80
C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
warnings.warn(
Number of images: 200
Aligning images
100%|████████████████████████████████████████████████████████████████████████████████| 200/200 [00:02<00:00, 94.06it/s]
100%|████████████████████████████████████████████████████████████████████████████████| 200/200 [00:04<00:00, 45.72it/s]
Aligning completed
Loading e4e over the pSp framework from checkpoint: ./pretrained_models/e4e_ffhq_encode.pt
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
Loading model from: C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\lpips\weights\v0.1\alex.pth
Calculating initial inversions
100%|████████████████████████████████████████████████████████████████████████████████| 200/200 [00:31<00:00, 6.27it/s]
Fine tuning generator
0%| | 0/80 [00:00<?, ?it/s]Setting up PyTorch plugin "bias_act_plugin"... Failed!
C:\Users\...\STIT\torch_utils\ops\bias_act.py:50: UserWarning: Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:
Traceback (most recent call last):
File "C:\Users\...\STIT\torch_utils\ops\bias_act.py", line 48, in _init
_plugin = custom_ops.get_plugin('bias_act_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
File "C:\Users\...\STIT\torch_utils\custom_ops.py", line 110, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1124, in load
return _jit_compile(
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1337, in _jit_compile
_write_ninja_file_and_build_library(
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1418, in _write_ninja_file_and_build_library
verify_ninja_availability()
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1474, in verify_ninja_availability
raise RuntimeError("Ninja is required to load C++ extensions")
RuntimeError: Ninja is required to load C++ extensions
warnings.warn('Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())
Setting up PyTorch plugin "upfirdn2d_plugin"... Failed!
C:\Users\...\STIT\torch_utils\ops\upfirdn2d.py:34: UserWarning: Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:
Traceback (most recent call last):
File "C:\Users\...\STIT\torch_utils\ops\upfirdn2d.py", line 32, in _init
_plugin = custom_ops.get_plugin('upfirdn2d_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
File "C:\Users\...\STIT\torch_utils\custom_ops.py", line 110, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1124, in load
return _jit_compile(
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1337, in _jit_compile
_write_ninja_file_and_build_library(
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1418, in _write_ninja_file_and_build_library
verify_ninja_availability()
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1474, in verify_ninja_availability
raise RuntimeError("Ninja is required to load C++ extensions")
RuntimeError: Ninja is required to load C++ extensions
warnings.warn('Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())
Setting up PyTorch plugin "upfirdn2d_plugin"... Failed!
C:\Users\...\STIT\torch_utils\ops\upfirdn2d.py:34: UserWarning: Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:
Traceback (most recent call last):
File "C:\Users\...\STIT\torch_utils\ops\upfirdn2d.py", line 32, in _init
_plugin = custom_ops.get_plugin('upfirdn2d_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
File "C:\Users\...\STIT\torch_utils\custom_ops.py", line 110, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1124, in load
return _jit_compile(
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1362, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "C:\Users\...\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\cpp_extension.py", line 1752, in _import_module_from_library
module = importlib.util.module_from_spec(spec)
File "<frozen importlib._bootstrap>", line 556, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1101, in create_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing upfirdn2d_plugin: The specified module could not be found.
...the same Traceback repeats indefinitely...
I also uninstalled and reinstalled torch to 1.10.2 after installing CUDA and cudnn 11.3
pip install ninja