Unable to extract features with pretrained models under conda
iSecret2048 opened this issue · comments
Hello,
we are trying to run the feature extraction part from the R2D2 project in our lab, but facing some issues with it.
We have set up everything under Anaconda, accourding to the project's Github readme file.
Each time we want to run the feature extraction script, we receive the following error:
(r2d2) geza@labor10-HP-Z440-Workstation:~/r2d2$ python extract.py --model models/r2d2_WAF_N16.pt --images /home/geza/Asztal/KiszomborEXT_Canon_vs_Canon_v3/Canon/001/resized/4R1A2625.png --top-k 5000
Launching on GPUs 0
>> Creating net = Quad_L2Net_ConfCFS()
( Model size: 486K parameters )
Extracting features for /home/geza/Asztal/KiszomborEXT_Canon_vs_Canon_v3/Canon/001/resized/4R1A2625.png
Traceback (most recent call last):
File "extract.py", line 182, in <module>
extract_keypoints(args)
File "extract.py", line 143, in extract_keypoints
verbose = True)
File "extract.py", line 97, in extract_multiscale
img = F.interpolate(img, (nh,nw), mode='bilinear', align_corners=False)
File "/home/geza/anaconda3/envs/r2d2/lib/python3.6/site-packages/torch/nn/functional.py", line 3013, in interpolate
scale_factor_list[0], scale_factor_list[1])
RuntimeError: CUDA error: no kernel image is available for execution on the device
Our regular environment consists of an HP workstation, equipped with an NVIDIA Tesla K20c GPU.
Here are some environment info:
Architecture: x86_64
CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
RAM: 32 GB
OS: Ubuntu 18.04.1 LTS (Bionic Beaver)
GPU: Tesla K20c and GeForce GT 710
NVIDIA driver info + SMI output:
cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 418.87.00 Thu Aug 8 15:35:46 CDT 2019
GCC version: gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
nvidia-smi
Wed Jul 1 10:53:20 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20c On | 00000000:05:00.0 Off | 0 |
| 30% 37C P8 17W / 225W | 0MiB / 4743MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GT 710 On | 00000000:06:00.0 N/A | N/A |
| 50% 53C P8 N/A / N/A | 104MiB / 973MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 1 Not Supported |
+-----------------------------------------------------------------------------+
Conda environment information:
conda info
active environment : r2d2
active env location : /home/geza/anaconda3/envs/r2d2
shell level : 1
user config file : /home/geza/.condarc
populated config files : /home/geza/.condarc
conda version : 4.8.3
conda-build version : 3.18.9
python version : 3.7.4.final.0
virtual packages : __cuda=10.1
__glibc=2.27
base environment : /home/geza/anaconda3 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/geza/anaconda3/pkgs
/home/geza/.conda/pkgs
envs directories : /home/geza/anaconda3/envs
/home/geza/.conda/envs
platform : linux-64
user-agent : conda/4.8.3 requests/2.23.0 CPython/3.7.4 Linux/4.15.0-108-generic ubuntu/18.04.1 glibc/2.27
UID:GID : 1010:1010
netrc file : None
offline mode : False
Installed packages with version:
# packages in environment at /home/geza/anaconda3/envs/r2d2:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
blas 1.0 mkl
ca-certificates 2020.6.24 0
certifi 2020.6.20 py36_0
cudatoolkit 10.1.243 h6bb024c_0
cycler 0.10.0 py36_0
dbus 1.13.16 hb2f20db_0
expat 2.2.9 he6710b0_2
fontconfig 2.13.0 h9420a91_0
freetype 2.10.2 h5ab3b9f_0
glib 2.65.0 h3eb4bd4_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb31296c_0
icu 58.2 he6710b0_3
intel-openmp 2020.1 217
jpeg 9b h024ee3a_2
kiwisolver 1.2.0 py36hfd86e86_0
ld_impl_linux-64 2.33.1 h53a641e_7
libedit 3.1.20191231 h7b6447c_0
libffi 3.3 he6710b0_1
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libpng 1.6.37 hbc83047_0
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 h2733197_1
libuuid 1.0.3 h1bed415_2
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 he19cac6_1
lz4-c 1.9.2 he6710b0_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py36hef1b27d_0
mkl 2020.1 217
mkl-service 2.3.0 py36he904b0f_0
mkl_fft 1.1.0 py36h23d657b_0
mkl_random 1.1.1 py36h0573a6f_0
ncurses 6.2 he6710b0_1
ninja 1.9.0 py36hfd86e86_0
numpy 1.18.5 py36ha1c710e_0
numpy-base 1.18.5 py36hde5b4d6_0
olefile 0.46 py36_0
openssl 1.1.1g h7b6447c_0
pcre 8.44 he6710b0_0
pillow 7.1.2 py36hb39fc2d_0
pip 20.1.1 py36_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py36h05f1152_2
python 3.6.10 h7579374_2
python-dateutil 2.8.1 py_0
pytorch 1.5.1 py3.6_cuda10.1.243_cudnn7.6.3_0 pytorch
qt 5.9.7 h5867ecd_1
readline 8.0 h7b6447c_0
scipy 1.5.0 py36h0b6359f_0
setuptools 47.3.1 py36_0
sip 4.19.8 py36hf484d3e_0
six 1.15.0 py_0
sqlite 3.32.3 h62c20be_0
tk 8.6.10 hbc83047_0
torchvision 0.6.1 py36_cu101 pytorch
tornado 6.0.4 py36h7b6447c_1
tqdm 4.47.0 py_0
wheel 0.34.2 py36_0
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.4 h0b5b093_3
The CUDA toolkit has been recently upgraded from version 10.0 to version 10.1. When the versions were inaccurate, we were receiving errors too:
Traceback (most recent call last):
File "extract.py", line 182, in <module>
extract_keypoints(args)
File "extract.py", line 112, in extract_keypoints
iscuda = common.torch_set_gpu(args.gpu)
File "/home/geza/r2d2/tools/common.py", line 32, in torch_set_gpu
os.environ['HOSTNAME'],os.environ['CUDA_VISIBLE_DEVICES'])
File "/home/geza/anaconda3/envs/r2d2/lib/python3.6/os.py", line 669, in __getitem__
raise KeyError(key) from None
KeyError: 'HOSTNAME'
We tried to run this on two machines (workstations) and the errors were the same. The configuration of both machines is the same.
Is here something that we're missing out?
Thank you for your answer in advance!
Hi @iSecret2048
This is because pytorch has stopped supporting 'old' GPUs like your K20c (see pytorch/pytorch#30532 ) Basically, this type of GPU lacks certain functionalities that are now implemented on all recent GPUs.
You can try to install an older version of pytorch or CUDA tool kit, maybe? I'm not sure which one you need.