tensorflow.python.framework.errors_impl.InvalidArgumentError
Karobben opened this issue · comments
I think I am ok with the environment = = but something happened and I can't run the demo
The code below is the result from the terminal. I think there is some compactable issue with TensorFlow.
Does anyone know how to solve this?
Python 3.8.10 (default, Jun 4 2021, 15:09:15)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from deepposekit.models import load_model
>>> from deepposekit.io import VideoReader
>>> model = load_model('/mnt/8A26661926660713/Github/DeepPoseKit/DeepPoseKit-Data/datasets/fly/best_model_densenet.h5')
2022-01-02 01:50:02.251177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2022-01-02 01:50:02.295954: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.296363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2200 computeCapability: 5.2
coreClock: 1.036GHz coreCount: 8 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 82.08GiB/s
2022-01-02 01:50:02.296873: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2022-01-02 01:50:02.299370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2022-01-02 01:50:02.301227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2022-01-02 01:50:02.301728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2022-01-02 01:50:02.303646: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2022-01-02 01:50:02.304828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2022-01-02 01:50:02.308512: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-01-02 01:50:02.308683: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.309068: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.309318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2022-01-02 01:50:02.309559: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-01-02 01:50:02.334861: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3099995000 Hz
2022-01-02 01:50:02.335524: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fdbdc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-01-02 01:50:02.335564: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2022-01-02 01:50:02.386637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.386997: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558b0617acb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-01-02 01:50:02.387020: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Quadro M2200, Compute Capability 5.2
2022-01-02 01:50:02.387317: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.387730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2200 computeCapability: 5.2
coreClock: 1.036GHz coreCount: 8 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 82.08GiB/s
2022-01-02 01:50:02.387837: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2022-01-02 01:50:02.387865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2022-01-02 01:50:02.387888: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2022-01-02 01:50:02.387910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2022-01-02 01:50:02.387933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2022-01-02 01:50:02.387955: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2022-01-02 01:50:02.387976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-01-02 01:50:02.388066: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.388546: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.388808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2022-01-02 01:50:02.388854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2022-01-02 01:50:02.389252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-01-02 01:50:02.389266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2022-01-02 01:50:02.389273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2022-01-02 01:50:02.389386: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.389898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:50:02.390179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1416 MB memory) -> physical GPU (device: 0, name: Quadro M2200, pci bus id: 0000:01:00.0, compute capability: 5.2)
>>> reader = VideoReader('/mnt/8A26661926660713/Github/DeepPoseKit/DeepPoseKit-Data/datasets/fly/video.avi')
>>> predictions = model.predict(reader)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper
return method(self, *args, **kwargs)
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict
tmp_batch_outputs = predict_function(iterator)
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(*args, **kwds)
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 650, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call
return self._call_flat(
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 593, in call
outputs = execute.execute(
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Must provide as many biases as the channel dimension of the input tensor: [3] vs. 9 in [1,9,192,192]
[[node StackedDenseNet/StackedDenseNet/image_net_preprocess/BiasAdd (defined at /mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/deepposekit/models/layers/deeplabcut.py:86) ]] [Op:__inference_predict_function_18964]
Errors may have originated from an input operation.
Input Source operations connected to node StackedDenseNet/StackedDenseNet/image_net_preprocess/BiasAdd:
StackedDenseNet/StackedDenseNet/image_net_preprocess/Const (defined at /mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/deepposekit/models/layers/deeplabcut.py:78)
StackedDenseNet/StackedDenseNet/image_net_preprocess/truediv (defined at /mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/deepposekit/models/layers/deeplabcut.py:62)
Function call stack:
predict_function
>>> exit()
2022-01-02 01:52:47.628144: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]
^[[A
╭─ ~ ·········································································· ✔ took 2m 59s flybase at 01:52:48
╰─ python
Python 3.8.10 (default, Jun 4 2021, 15:09:15)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from deepposekit.models import load_model
>>> from deepposekit.io import VideoReader
>>> model = load_model('/mnt/8A26661926660713/Github/DeepPoseKit/DeepPoseKit-Data/datasets/fly/best_model_densenet.h5')
2022-01-02 01:53:21.748213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2022-01-02 01:53:21.786445: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.786790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2200 computeCapability: 5.2
coreClock: 1.036GHz coreCount: 8 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 82.08GiB/s
2022-01-02 01:53:21.787363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2022-01-02 01:53:21.789731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2022-01-02 01:53:21.791381: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2022-01-02 01:53:21.791733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2022-01-02 01:53:21.793240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2022-01-02 01:53:21.794155: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2022-01-02 01:53:21.797057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-01-02 01:53:21.797216: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.797532: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.797756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2022-01-02 01:53:21.797982: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-01-02 01:53:21.822824: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3099995000 Hz
2022-01-02 01:53:21.823443: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f942c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-01-02 01:53:21.823484: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2022-01-02 01:53:21.866443: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.866868: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d1015b7e80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-01-02 01:53:21.866884: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Quadro M2200, Compute Capability 5.2
2022-01-02 01:53:21.867089: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.867352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2200 computeCapability: 5.2
coreClock: 1.036GHz coreCount: 8 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 82.08GiB/s
2022-01-02 01:53:21.867413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2022-01-02 01:53:21.867430: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2022-01-02 01:53:21.867444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2022-01-02 01:53:21.867459: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2022-01-02 01:53:21.867474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2022-01-02 01:53:21.867488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2022-01-02 01:53:21.867504: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-01-02 01:53:21.867567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.867824: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.868032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2022-01-02 01:53:21.868062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2022-01-02 01:53:21.868393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-01-02 01:53:21.868403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2022-01-02 01:53:21.868409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2022-01-02 01:53:21.868540: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.868870: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-02 01:53:21.869108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1409 MB memory) -> physical GPU (device: 0, name: Quadro M2200, pci bus id: 0000:01:00.0, compute capability: 5.2)
>>> reader = VideoReader('/mnt/8A26661926660713/Github/DeepPoseKit/DeepPoseKit-Data/datasets/fly/video.avi')
>>> predictions = model.predict(reader)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper
return method(self, *args, **kwargs)
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict
tmp_batch_outputs = predict_function(iterator)
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(*args, **kwds)
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 650, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call
return self._call_flat(
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 593, in call
outputs = execute.execute(
File "/mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Must provide as many biases as the channel dimension of the input tensor: [3] vs. 9 in [1,9,192,192]
[[node StackedDenseNet/StackedDenseNet/image_net_preprocess/BiasAdd (defined at /mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/deepposekit/models/layers/deeplabcut.py:88) ]] [Op:__inference_predict_function_18964]
Errors may have originated from an input operation.
Input Source operations connected to node StackedDenseNet/StackedDenseNet/image_net_preprocess/BiasAdd:
StackedDenseNet/StackedDenseNet/image_net_preprocess/truediv (defined at /mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/deepposekit/models/layers/deeplabcut.py:64)
StackedDenseNet/StackedDenseNet/image_net_preprocess/Const (defined at /mnt/8A26661926660713/Conda/miniconda/envs/flybase/lib/python3.8/site-packages/deepposekit/models/layers/deeplabcut.py:80)
Function call stack:
predict_function
By the way,
my TensorFlow version.
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
tensorflow 2.2.3
tensorflow-estimator 2.2.0
tensorflow-gpu 2.2.0
tensorflow-io-gcs-filesystem 0.22.0
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
This issue has been automatically closed because it has not had recent activity.