When running the gtav_inference_demo.bat it gets stuck.
garrethcline opened this issue · comments
Not sure why but when running the demo bat file it seems to get stuck.
(base) C:\Users\lanzc\Documents\GitHub\GANTheftAuto>python inference.py --saved_model ./trained_models/gan_5_1_17.pt --data gtav:./data/gtav/gtagan_2_sample --inference_image_path ./data/gtav/2.png --show_base_images True --upsample_model ./trained_models/upsample---[_20]---[______3171]---[_____63420].h5
Adding attention layer in G at resolution 32
Adding attention layer in G at resolution 64
Param count for Gs initialized parameters: 29664579
Adding attention layer in D at resolution 64
Adding attention layer in D at resolution 32
Param count for Ds initialized parameters: 1081733
It stays here forever.
Can you please provide info about:
- GPU that you are using
- Python version
- CUDA version
- cuDNN version
?
GPU: 3090 turbo 24g
| NVIDIA-SMI 466.47 Driver Version: 466.47 CUDA Version: 11.3 |
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
| 0 NVIDIA GeForce ... WDDM | 00000000:0A:00.0 On | N/A |
| 75% 54C P0 130W / 350W | 1463MiB / 24576MiB | 2% Default |
| | | N/A |
CUDA: nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:41:42_Pacific_Daylight_Time_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0
Python: python --version
Python 3.8.5
cuDNN: cudnn-11.3-windows-x64-v8.2.1.32
11.3
Now it seems to say
2021-06-19 19:51:53.823268: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-19 19:51:54.257639: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-19 19:51:55.022692: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2021-06-19 19:51:55.024758: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2021-06-19 19:51:55.026723: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2021-06-19 19:51:55.028838: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
File "inference.py", line 273, in
inference(opts.gpu, opts)
File "inference.py", line 166, in inference
upsampled_img = upsample.inference(np.rollaxis(prev_state[0].cpu().numpy(), 0, 3))
File "C:\Users\lanzc\Documents\GitHub\GANTheftAuto\upsample\upsample.py", line 17, in inference
upsampled = model.predict(np.expand_dims(img, axis=0))
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1727, in predict
tmp_batch_outputs = self.predict_function(iterator)
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 889, in call
result = self._call(*args, **kwds)
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 956, in _call
return self._concrete_stateful_fn._call_flat(
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1960, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 591, in call
outputs = execute.execute(
File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node model/spectral_normalization_1/conv2d_1/Conv2D (defined at C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_addons\layers\spectral_normalization.py:100) ]] [Op:__inference_predict_function_3325]
Errors may have originated from an input operation.
Input Source operations connected to node model/spectral_normalization_1/conv2d_1/Conv2D:
model/up_sampling2d/resize/ResizeNearestNeighbor (defined at C:\Users\lanzc\Documents\GitHub\GANTheftAuto\upsample\upsample.py:17)
Function call stack:
predict_function
Was missing some windows environment paths before.
Going to try to reinstall CuDNN for 8.1.0.
Yes, cuDNN 8.1 or newer is required with Tensorflow 2.5. In case of issues, you can use TF 2.4.