Model conversion error
MyraBaba opened this issue · comments
root@84a3a2a7c8f3:/app# python3 prepare_models.py
mxnet version: 1.6.0
onnx version: 1.7.0
[12:45:13] INFO - Preparing 'arcface_r100_v1' model...
Downloading model: arcface_r100_v1...
Model file is not found. Downloading.
Downloading /models/mxnet/arcface_r100_v1.zip from http://insightface.ai/files/models/arcface_r100_v1.zip...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 237710/237710 [03:47<00:00, 1046.77KB/s]
[12:49:02] INFO - Creating intermediate copy of source model...
[12:49:03] INFO - Applying RetinaFace specific fixes to input MXNet model before conversion...
[12:49:03] INFO - Exporting to ONNX...
[12:49:03] INFO - Converting json and weight file to sym and params
[12:49:03] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.0.0. Attempting to upgrade...
[12:49:03] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[12:49:11] INFO - Input shape of the model [(1, 3, 112, 112)]
[12:49:11] INFO - Exported ONNX file /models/onnx/arcface_r100_v1/arcface_r100_v1.onnx saved to disk
[12:49:11] INFO - Applying ArcFace specific fixes to output ONNX
[12:49:14] INFO - Removing initializer from inputs in ONNX model...
[12:49:14] INFO - Removing intermediate *.symbol and *.params
[12:49:14] INFO - Building TRT engine for arcface_r100_v1...
Traceback (most recent call last):
File "prepare_models.py", line 45, in <module>
prepare_models()
File "prepare_models.py", line 40, in prepare_models
prepare_backend(model_name=model, backend_name=backend_name, im_size=max_size, force_fp16=force_fp16,
File "/app/modules/model_zoo/getter.py", line 127, in prepare_backend
convert_onnx(temp_onnx_model,
TypeError: 'NoneType' object is not callable
Hi, @MyraBaba ! Unfortunately I wasn't able to reproduce your error using clean docker build without cache and removing previously converted models.
@SthPhoenix I delete all and make clean install now all is worked.
One small issue is I try to run without docker than I noticed system doesnt work Python3,7 only 3.8 right?
I can't recall using any 3.8 specific features, but it's possible that some of dependencies require python 3.8
I have some warning about converting
Retinaface_r50_v1 to trt float16
root@8e0358120280:/app# python3 prepare_models.py
mxnet version: 1.6.0
onnx version: 1.7.0
[07:36:42] INFO - Preparing 'glint360k_r100FC_1.0' model...
[07:36:42] INFO - Preparing 'retinaface_r50_v1' model...
Downloading model: retinaface_r50_v1...
Model file is not found. Downloading.
Downloading /models/mxnet/retinaface_r50_v1.zip from http://insightface.ai/files/models/retinaface_r50_v1.zip...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107062/107062 [00:58<00:00, 1840.32KB/s]
[07:37:42] INFO - Creating intermediate copy of source model...
[07:37:42] INFO - Applying RetinaFace specific fixes to input MXNet model before conversion...
[07:37:42] INFO - Exporting to ONNX...
[07:37:42] INFO - Converting json and weight file to sym and params
[07:37:42] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.3.0. Attempting to upgrade...
[07:37:42] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[07:37:45] WARNING - Using an experimental ONNX operator: Crop. Its definition can change.
[07:37:45] WARNING - Using an experimental ONNX operator: Crop. Its definition can change.
[07:37:47] INFO - Input shape of the model [(1, 3, 640, 640)]
[07:37:47] INFO - Exported ONNX file /models/onnx/retinaface_r50_v1/retinaface_r50_v1.onnx saved to disk
[07:37:47] INFO - Applying ArcFace specific fixes to output ONNX
[07:37:49] INFO - Removing initializer from inputs in ONNX model...
[07:37:50] INFO - Removing intermediate *.symbol and *.params
[07:37:50] INFO - Reshaping ONNX inputs to: (1, 3, 640, 640)
[07:37:50] INFO - Building TRT engine for retinaface_r50_v1...
[07:38:03] INFO - Building TensorRT engine with FP16 support.
[TensorRT] WARNING: /home/jenkins/workspace/OSS/L0_MergeRequest/oss/parsers/onnx/onnx2trt_utils.cpp:226: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] WARNING: No implementation obeys reformatting-free rules, at least 10 reformatting nodes are needed, now picking the fastest path instead.
[07:41:24] INFO - Building TRT engine complete!
[07:41:24] INFO - Preparing 'genderage_v1' model...
Downloading model: genderage_v1...
Model file is not found. Downloading.
Downloading /models/mxnet/genderage_v1.zip from http://insightface.ai/files/models/genderage_v1.zip...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 987/987 [00:01<00:00, 855.40KB/s]
[07:41:26] INFO - Creating intermediate copy of source model...
[07:41:26] INFO - Applying RetinaFace specific fixes to input MXNet model before conversion...
[07:41:26] INFO - Exporting to ONNX...
[07:41:26] INFO - Converting json and weight file to sym and params
[07:41:26] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.3.0. Attempting to upgrade...
[07:41:26] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[07:41:26] INFO - Input shape of the model [(1, 3, 112, 112)]
[07:41:26] INFO - Exported ONNX file /models/onnx/genderage_v1/genderage_v1.onnx saved to disk
[07:41:26] INFO - Applying ArcFace specific fixes to output ONNX
[07:41:26] INFO - Removing initializer from inputs in ONNX model...
[07:41:26] INFO - Removing intermediate *.symbol and *.params
[07:41:26] INFO - Building TRT engine for genderage_v1...
[07:41:26] INFO - Building TensorRT engine with FP16 support.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: Try increasing the workspace size to 4194304 bytes to get better performance.
[TensorRT] WARNING: No implementation obeys reformatting-free rules, at least 2 reformatting nodes are needed, now picking the fastest path instead.
[07:43:14] INFO - Building TRT engine complete!
And I see some deterioration in quality detection.
Is it fixible?)
Workspace could be increased.
Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. might be fixed by converting mxnet weights to fp32 during ONNX conversion, but I believe it wold have same results as TRT automatic cast down.
No implementation obeys reformatting-free rules, at least 2 reformatting nodes are needed, now picking the fastest path instead. This warning is specific to fp16 inference, and I have to admit I have no idea what does it mean yet.
I believe if you see some quality drop during detection it is mostly caused by fp16 inference, though I haven't noticed any significant changes, except for bboxes and landmarks sometimes being different for about 1-2 pixels compared to fp32 Inference. If you have any examples to share, that would be great for debugging.
Closing for now as resolved, since above warnings doesn't affect usability.