[AOT]Model Compiler shows error for tf2onnx converted onnx MobileNetV2 model
angelb147 opened this issue · comments
Hello,I'm a beginner in the Glow platform and I tried to make an executable for the mobilenet_v2_130_224 pretrained model in
https://tfhub.dev/google/imagenet/mobilenet_v2_130_224/classification/5 with the instructions mentioned here https://github.com/pytorch/glow/blob/master/docs/AOT.md.
While the MobilenetV1 example works fine for me(both the tf2onnx step and the model compiler one),when I try the same on the MobileNetV2 model above,while the tf2onnx step works and the onnx is created, I then get the following error on the model-compiler step.I apologize in advance if the issue post is messy,it's the first one I post on Github in general.Thank you for your time.
- Model Compiler for MobileNetV2 Model with Error
./model-compiler -backend=CPU -model=/home/ampampos/tries/mobilenetv2/imagenet_mobilenet_v2_130_224_classification_5/output/model.onnx -onnx-define-symbol=unk__670,1 -emit-bundle=bundle
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0410 09:23:05.847636 4098772 Error.cpp:123] exitOnError(Error) got an unexpected ErrorValue:
Error message: Node has no ints attribute with this name
Error return stack:
--------------------------------------------------------------------------------
/home/ampampos/tries/glow/include/glow/Importer/ProtobufLoader.h:95
--------------------------------------------------------------------------------
/home/ampampos/tries/glow/include/glow/Importer/ProtobufLoader.h:179
--------------------------------------------------------------------------------
/home/ampampos/tries/glow/lib/Importer/ONNXModelLoader.cpp:2935
--------------------------------------------------------------------------------
/home/ampampos/tries/glow/lib/Importer/ONNXModelLoader.cpp:6270
--------------------------------------------------------------------------------
/home/ampampos/tries/glow/lib/Importer/ONNXModelLoader.cpp:6402
--------------------------------------------------------------------------------
/home/ampampos/tries/glow/lib/Importer/ONNXModelLoader.cpp:6442
--------------------------------------------------------------------------------
/home/ampampos/tries/glow/lib/Importer/ONNXModelLoader.cpp:6449
--------------------------------------------------------------------------------
*** Check failure stack trace: ***
./model-compiler[0xc21c7e]
./model-compiler[0xc1ff01]
./model-compiler[0xc22084]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x143bf)[0x7f5f739223bf]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f5f7341103b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12a)[0x7f5f733f0858]
/lib/x86_64-linux-gnu/libglog.so.0(+0x9e4b)[0x7f5f743bfe4b]
/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google10LogMessage4FailEv+0x12)[0x7f5f743c31c2]
/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google10LogMessage9SendToLogEv+0x26a)[0x7f5f743c825a]
/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google10LogMessage5FlushEv+0xbe)[0x7f5f743c2ebe]
/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google15LogMessageFatalD2Ev+0xe)[0x7f5f743c36ee]
./model-compiler[0x2ba2069]
./model-compiler[0x9ef22a]
./model-compiler[0x4b9061]
./model-compiler[0x4f75bf]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf2)[0x7f5f733f20b2]
./model-compiler[0x4b7dfd]
Aborted (core dumped)
For extra context:
- tf2onnx Conversion for MobileNetV2 model
python3 -m tf2onnx.convert --saved-model /home/ampampos/tries/mobilenetv2/imagenet_mobilenet_v2_130_224_classification_5/ --output /home/ampampos/tries/mobilenetv2/imagenet_mobilenet_v2_130_224_classification_5/output/model.onnx --opset 13
2022-04-10 09:20:04.600715: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-04-10 09:20:04.600741: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
2022-04-10 09:20:05.563505: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-04-10 09:20:05.563530: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-04-10 09:20:05.563563: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (sheldon): /proc/driver/nvidia/version does not exist
2022-04-10 09:20:05.563786: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-04-10 09:20:05,564 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2022-04-10 09:20:06,506 - INFO - Signatures found in model: [serving_default].
2022-04-10 09:20:06,506 - WARNING - '--signature_def' not specified, using first signature: serving_default
2022-04-10 09:20:06,507 - INFO - Output names: ['logits']
2022-04-10 09:20:06.519930: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2022-04-10 09:20:06.520076: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-04-10 09:20:06.592896: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1164] Optimization results for grappler item: graph_to_optimize
function_optimizer: Graph size after: 1384 nodes (1118), 1925 edges (1659), time = 23.974ms.
function_optimizer: function_optimizer did nothing. time = 0.019ms.
WARNING:tensorflow:From /home/ampampos/.local/lib/python3.8/site-packages/tf2onnx/tf_loader.py:711: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2022-04-10 09:20:07,670 - WARNING - From /home/ampampos/.local/lib/python3.8/site-packages/tf2onnx/tf_loader.py:711: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2022-04-10 09:20:07.698880: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2022-04-10 09:20:07.699027: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-04-10 09:20:07.885707: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1164] Optimization results for grappler item: graph_to_optimize
constant_folding: Graph size after: 782 nodes (-524), 1058 edges (-786), time = 99.405ms.
function_optimizer: function_optimizer did nothing. time = 0.004ms.
constant_folding: Graph size after: 782 nodes (0), 1058 edges (0), time = 42.883ms.
function_optimizer: function_optimizer did nothing. time = 0.004ms.
2022-04-10 09:20:08,131 - INFO - Using tensorflow=2.8.0, onnx=1.11.0, tf2onnx=1.10.0/cd9f14
2022-04-10 09:20:08,131 - INFO - Using opset <onnx, 13>
2022-04-10 09:20:10,095 - INFO - Computed 0 values for constant folding
2022-04-10 09:20:11,491 - INFO - Optimizing ONNX model
2022-04-10 09:20:12,617 - INFO - After optimization: BatchNormalization -45 (52->7), Const -220 (352->132), Identity -95 (95->0), Reshape -17 (17->0), Transpose -228 (229->1)
2022-04-10 09:20:12,650 - INFO -
2022-04-10 09:20:12,651 - INFO - Successfully converted TensorFlow model /home/ampampos/tries/mobilenetv2/imagenet_mobilenet_v2_130_224_classification_5/ to ONNX
2022-04-10 09:20:12,651 - INFO - Model inputs: ['inputs']
2022-04-10 09:20:12,651 - INFO - Model outputs: ['logits']
2022-04-10 09:20:12,651 - INFO - ONNX model is saved at /home/ampampos/tries/mobilenetv2/imagenet_mobilenet_v2_130_224_classification_5/output/model.onnx
And the Linux Bash messages for the MobileNetV1 example
- tf2onnx Conversion for MobileNetV1 Example
python3 -m tf2onnx.convert \
> --input /home/ampampos/tries/mobiletestlinux/mobilenet_v1_0.25_128/mobilenet_v1_0.25_128_frozen.pb \
> --inputs input:0 \
> --outputs MobilenetV1/Predictions/Softmax:0 \
> --output mobilenet_v1_0.25_128.onnx
2022-04-10 09:09:45.291983: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-04-10 09:09:45.292009: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
2022-04-10 09:09:46.257039: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-04-10 09:09:46.257064: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-04-10 09:09:46.257083: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (sheldon): /proc/driver/nvidia/version does not exist
2022-04-10 09:09:46.257298: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From /home/ampampos/.local/lib/python3.8/site-packages/tf2onnx/tf_loader.py:305: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
2022-04-10 09:09:46,302 - WARNING - From /home/ampampos/.local/lib/python3.8/site-packages/tf2onnx/tf_loader.py:305: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/convert_to_constants.py:925: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2022-04-10 09:09:46,302 - WARNING - From /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/convert_to_constants.py:925: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2022-04-10 09:09:46.385232: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2022-04-10 09:09:46.385388: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-04-10 09:09:46.436461: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1164] Optimization results for grappler item: graph_to_optimize
constant_folding: Graph size after: 227 nodes (-137), 226 edges (-137), time = 11.232ms.
function_optimizer: function_optimizer did nothing. time = 0.008ms.
constant_folding: Graph size after: 227 nodes (0), 226 edges (0), time = 4.052ms.
function_optimizer: function_optimizer did nothing. time = 0.002ms.
2022-04-10 09:09:46,462 - INFO - Using tensorflow=2.8.0, onnx=1.11.0, tf2onnx=1.10.0/cd9f14
2022-04-10 09:09:46,462 - INFO - Using opset <onnx, 9>
2022-04-10 09:09:46,591 - INFO - Computed 0 values for constant folding
2022-04-10 09:09:46,749 - INFO - Optimizing ONNX model
2022-04-10 09:09:47,041 - INFO - After optimization: BatchNormalization -27 (27->0), Const -95 (151->56), Identity -3 (3->0), Reshape -13 (13->0), Transpose -124 (125->1)
2022-04-10 09:09:47,047 - INFO -
2022-04-10 09:09:47,047 - INFO - Successfully converted TensorFlow model /home/ampampos/tries/mobiletestlinux/mobilenet_v1_0.25_128/mobilenet_v1_0.25_128_frozen.pb to ONNX
2022-04-10 09:09:47,047 - INFO - Model inputs: ['input:0']
2022-04-10 09:09:47,047 - INFO - Model outputs: ['MobilenetV1/Predictions/Softmax:0']
2022-04-10 09:09:47,047 - INFO - ONNX model is saved at mobilenet_v1_0.25_128.onnx
- Model Compiler for MobileNetV1 Example
./model-compiler -backend=CPU -model=/home/ampampos/tries/mobiletestlinux/mobilenet_v1_0.25_128/mobilenet_v1_0.25_128.onnx -onnx-define-symbol=unk__282,1 -emit-bundle=bundle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0410 09:17:19.378022 4097903 HostManager.cpp:239] Destroying host manager...
Hey, even I am facing the same issue, did you resolve it?