Error while create postprocess for onnx yolov7

Question

Error while create postprocess for onnx yolov7

HoangTienDuc opened this issue 10 months ago · comments

Jeff Hoang commented 10 months ago

Issue Type

Bug

OS

Ubuntu

OS architecture

x86_64

Programming Language

Python

Framework

TensorFlow, TensorFlowLite

Model name and Weights/Checkpoints URL

Model name: yolov7
https://github.com/PINTO0309/PINTO_model_zoo/tree/18e75913cb97eefa2d8fe9f87b86841407b4c826/307_YOLOv7/post_process_gen_tools

Description

Hi PINTO, your work is awesome.
I try to replicate your work to post-process my custon yolov7 model.
First, i tried to follow the instruction from your model zoo yolov7 post-process without any change, but with your original postprocess docker (1.1.5, 1.1.15), i cannot "update_model_dims.update_inputs_outputs_dims".
Tried with diffirent dockers, tf2onnx, tensorflow version, but it is still not work.
Can you pls give me some advice? Thanks

Relevant Log Output

2023-08-19 04:45:55,743 - INFO - Using tensorflow=2.13.0, onnx=1.14.0, tf2onnx=1.12.1/b6d590
2023-08-19 04:45:55,743 - INFO - Using opset <onnx, 11>
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
2023-08-19 04:45:55,764 - WARNING - Error loading model into tflite interpreter: _get_tensor_details() missing 1 required positional argument: 'subgraph_index'
2023-08-19 04:45:55,770 - INFO - Optimizing ONNX model
2023-08-19 04:45:55,784 - INFO - After optimization: Cast -1 (1->0), Const -2 (6->4), Identity -1 (1->0)
2023-08-19 04:45:55,785 - INFO - 
2023-08-19 04:45:55,785 - INFO - Successfully converted TensorFlow model saved_model_postprocess/nms_score_gather_nd.tflite to ONNX
2023-08-19 04:45:55,785 - INFO - Model inputs: ['serving_default_input_1:0', 'serving_default_input_2:0']
2023-08-19 04:45:55,785 - INFO - Model outputs: ['PartitionedCall:0']
2023-08-19 04:45:55,785 - INFO - ONNX model is saved at nms_score_gather_nd.onnx
INFO: Finish!
INFO: Finish!
INFO: Finish!
INFO: Finish!
OUTPUT_MODEL_PATH:  nms_score_gather_nd.onnx
model.graph.input:  [name: "serving_default_input_1:0"
type {
  tensor_type {
    elem_type: 1
    shape {
      dim {
        dim_value: 1
      }
      dim {
        dim_value: 80
      }
      dim {
        dim_value: 5040
      }
    }
  }
}
, name: "serving_default_input_2:0"
type {
  tensor_type {
    elem_type: 7
    shape {
      dim {
        dim_param: "unk__10"
      }
      dim {
        dim_value: 3
      }
    }
  }
}
]
input_dicts:  {'gn_scores': ['1', '80', '5040'], 'gn_selected_indices': ['N', '3']}
output_dicts:  {'final_scores': ['N', '1']}
input_dims:  {'gn_scores': ['1', '80', '5040'], 'gn_selected_indices': ['N', '3']}
Traceback (most recent call last):
  File "make_input_output_shape_update.py", line 75, in <module>
    updated_model = update_model_dims.update_inputs_outputs_dims(
  File "/home/user/.local/lib/python3.8/site-packages/onnx/tools/update_model_dims.py", line 88, in update_inputs_outputs_dims
    input_dim_arr = input_dims[input_name]
KeyError: 'serving_default_input_1:0'

URL or source code for simple inference testing code

No response

Jeff Hoang · Answer 1 · Sun Aug 20 2023 13:32:51 GMT+0800 (China Standard Time)

Hi Katsuya, Thanks for your reply. I only run the sample "convert_script.txt" from 307_yolov7/post_process_gen_tools Reproduce: ``` OPSET=11 BATCHES=1 BOXES=5040 CLASSES=80 ################################################### Boxes + Scores python make_boxes_scores.py -o ${OPSET} -b ${BATCHES} -x ${BOXES} -c ${CLASSES} python make_cxcywh_y1x1y2x2.py -o ${OPSET} -b ${BATCHES} -x ${BOXES} snc4onnx \ --input_onnx_file_paths boxes_scores_${BOXES}.onnx cxcywh_y1x1y2x2_${BOXES}.onnx \ --srcop_destop boxes_cxcywh cxcywh \ --output_onnx_file_path boxes_y1x1y2x2_scores_${BOXES}.onnx ################################################### NonMaxSuppression sog4onnx \ --op_type Constant \ --opset ${OPSET} \ --op_name max_output_boxes_per_class_const \ --output_variables max_output_boxes_per_class int64 [1] \ --attributes value int64 [20] \ --output_onnx_file_path Constant_max_output_boxes_per_class.onnx sog4onnx \ --op_type Constant \ --opset ${OPSET} \ --op_name iou_threshold_const \ --output_variables iou_threshold float32 [1] \ --attributes value float32 [0.5] \ --output_onnx_file_path Constant_iou_threshold.onnx sog4onnx \ --op_type Constant \ --opset ${OPSET} \ --op_name score_threshold_const \ --output_variables score_threshold float32 [1] \ --attributes value float32 [0.4] \ --output_onnx_file_path Constant_score_threshold.onnx OP=NonMaxSuppression LOWEROP=${OP,,} sog4onnx \ --op_type ${OP} \ --opset ${OPSET} \ --op_name ${LOWEROP}${OPSET} \ --input_variables boxes_var float32 [${BATCHES},${BOXES},4] \ --input_variables scores_var float32 [${BATCHES},${CLASSES},${BOXES}] \ --input_variables max_output_boxes_per_class_var int64 [1] \ --input_variables iou_threshold_var float32 [1] \ --input_variables score_threshold_var float32 [1] \ --output_variables selected_indices int64 [\'N\',3] \ --attributes center_point_box int64 0 \ --output_onnx_file_path ${OP}${OPSET}.onnx snc4onnx \ --input_onnx_file_paths Constant_max_output_boxes_per_class.onnx NonMaxSuppression11.onnx \ --srcop_destop max_output_boxes_per_class max_output_boxes_per_class_var \ --output_onnx_file_path NonMaxSuppression${OPSET}.onnx snc4onnx \ --input_onnx_file_paths Constant_iou_threshold.onnx NonMaxSuppression${OPSET}.onnx \ --srcop_destop iou_threshold iou_threshold_var \ --output_onnx_file_path NonMaxSuppression${OPSET}.onnx snc4onnx \ --input_onnx_file_paths Constant_score_threshold.onnx NonMaxSuppression${OPSET}.onnx \ --srcop_destop score_threshold score_threshold_var \ --output_onnx_file_path NonMaxSuppression${OPSET}.onnx soc4onnx \ --input_onnx_file_path NonMaxSuppression${OPSET}.onnx \ --output_onnx_file_path NonMaxSuppression${OPSET}.onnx \ --opset ${OPSET} ################################################### Boxes + Scores + NonMaxSuppression snc4onnx \ --input_onnx_file_paths boxes_y1x1y2x2_scores_${BOXES}.onnx NonMaxSuppression${OPSET}.onnx \ --srcop_destop scores scores_var y1x1y2x2 boxes_var \ --output_onnx_file_path nms_yolov7_${BOXES}.onnx ################################################### Myriad workaround Mul OP=Mul LOWEROP=${OP,,} OPSET=${OPSET} sog4onnx \ --op_type ${OP} \ --opset ${OPSET} \ --op_name ${LOWEROP}${OPSET} \ --input_variables workaround_mul_a int64 [\'N\',3] \ --input_variables workaround_mul_b int64 [1] \ --output_variables workaround_mul_out int64 [\'N\',3] \ --output_onnx_file_path ${OP}${OPSET}_workaround.onnx ############ Myriad workaround Constant sog4onnx \ --op_type Constant \ --opset ${OPSET} \ --op_name workaround_mul_const_op \ --output_variables workaround_mul_const int64 [1] \ --attributes value int64 [1] \ --output_onnx_file_path Constant_workaround_mul.onnx ############ Myriad workaround Mul + Myriad workaround Constant snc4onnx \ --input_onnx_file_paths Constant_workaround_mul.onnx Mul${OPSET}_workaround.onnx \ --srcop_destop workaround_mul_const workaround_mul_b \ --output_onnx_file_path Mul${OPSET}_workaround.onnx ################################################### NonMaxSuppression + Myriad workaround Mul snc4onnx \ --input_onnx_file_paths nms_yolov7_${BOXES}.onnx Mul${OPSET}_workaround.onnx \ --srcop_destop selected_indices workaround_mul_a \ --output_onnx_file_path nms_yolov7_${BOXES}.onnx ################################################### Score GatherND python make_score_gather_nd.py -b ${BATCHES} -x ${BOXES} -c ${CLASSES} python -m tf2onnx.convert \ --opset ${OPSET} \ --tflite saved_model_postprocess/nms_score_gather_nd.tflite \ --output nms_score_gather_nd.onnx sor4onnx \ --input_onnx_file_path nms_score_gather_nd.onnx \ --old_new ":0" "" \ --output_onnx_file_path nms_score_gather_nd.onnx sor4onnx \ --input_onnx_file_path nms_score_gather_nd.onnx \ --old_new "serving_default_input_1" "gn_scores" \ --output_onnx_file_path nms_score_gather_nd.onnx \ --mode inputs sor4onnx \ --input_onnx_file_path nms_score_gather_nd.onnx \ --old_new "serving_default_input_2" "gn_selected_indices" \ --output_onnx_file_path nms_score_gather_nd.onnx \ --mode inputs sor4onnx \ --input_onnx_file_path nms_score_gather_nd.onnx \ --old_new "PartitionedCall" "final_scores" \ --output_onnx_file_path nms_score_gather_nd.onnx \ --mode outputs python make_input_output_shape_update.py \ --input_onnx_file_path nms_score_gather_nd.onnx \ --output_onnx_file_path nms_score_gather_nd.onnx \ --input_names gn_scores \ --input_names gn_selected_indices \ --input_shapes ${BATCHES} ${CLASSES} ${BOXES} \ --input_shapes N 3 \ --output_names final_scores \ --output_shapes N 1 ```

…

On Sun, Aug 20, 2023 at 12:28 PM Katsuya Hyodo ***@***.***> wrote: KeyError: 'serving_default_input_1:0' is displayed. That's all there is to it. How would I know what model you are using? — Reply to this email directly, view it on GitHub <#358 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMWDRMRIZ42A7M5WQBQLSO3XWGOADANCNFSM6AAAAAA3WKOJDY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Katsuya Hyodo · Answer 2 · Sun Aug 20 2023 14:00:48 GMT+0800 (China Standard Time)

An error occurred during the conversion because the specification of sor4onnx, a homebrew tool, was changed. Committed to fix convert_script.txt.

This script is still buggy as it has never been maintained since it was first created on a trial basis.

The final output is Y1X1Y2X2.

Katsuya Hyodo · Answer 3 · Sun Aug 20 2023 14:11:50 GMT+0800 (China Standard Time)

In addition, the binding bug of final_box_nums was adjusted.

Katsuya Hyodo · Answer 4 · Sun Aug 20 2023 14:20:20 GMT+0800 (China Standard Time)

x1y1x2y2 -> y1x1y2x2

Katsuya Hyodo · Answer 5 · Sun Aug 20 2023 14:24:04 GMT+0800 (China Standard Time)

In the Concat just before NMS, we need to generate both the Y1X1Y2X2 tensor for NMS and the X1Y1X2Y2 for the final output, but it is a year old script and I have not modified it because it is too much trouble.

Katsuya Hyodo · Answer 6 · Sun Aug 20 2023 14:26:45 GMT+0800 (China Standard Time)

One more point. If you don't add Sqrt here, the score should be oddly small, less than half.

Jeff Hoang · Answer 7 · Sun Aug 20 2023 18:08:35 GMT+0800 (China Standard Time)

Thank you, I will check it and let you know later

…

On Sun, Aug 20, 2023 at 1:26 PM Katsuya Hyodo ***@***.***> wrote: One more point. If you don't add Sqrt here, the score should be oddly small, less than half. [image: image] <https://user-images.githubusercontent.com/33194443/261835292-0d170c79-da66-4e52-9253-3b878c23d273.png> — Reply to this email directly, view it on GitHub <#358 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMWDRMWJF3HNYUUATQZKWCTXWGU27ANCNFSM6AAAAAA3WKOJDY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Katsuya Hyodo · Answer 8 · Sun Aug 20 2023 18:34:40 GMT+0800 (China Standard Time)

All corrected.

Jeff Hoang · Answer 9 · Mon Aug 21 2023 15:47:20 GMT+0800 (China Standard Time)

Awesome, it works. Thank you very much. Your work is incredible. I hope in the future you will have more projects to help in the community.

…

On Sun, Aug 20, 2023 at 5:34 PM Katsuya Hyodo ***@***.***> wrote: All corrected. [image: image] <https://user-images.githubusercontent.com/33194443/261846206-ea8ceef2-1f6a-46e3-8986-c5024b1b3315.png> — Reply to this email directly, view it on GitHub <#358 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMWDRMT3DO67D675M2UX27DXWHR4VANCNFSM6AAAAAA3WKOJDY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Jeff Hoang · Answer 10 · Fri Sep 08 2023 00:36:23 GMT+0800 (China Standard Time)

Duplicate result from an image to other images

Hi @PINTO0309 , I convert the model from onnx to tensorrt, then use tensorrt code to make prediction.

For model yolov7_post_256x320.onnx provided from you, it works perfectly
For my custom model. Input size is 256x320, after using "post_process_gen_tools" I got the same model as you. However, after testing with a series of images, I got an error that duplicated results from one image to another.

For example:


image_path /data/data_video/24/pgie/images/0_10-8-2023-17-46_11_109380.png, batch_pred: [[array([149,  66, 207, 156], dtype=int32), array([175, 290, 213, 320], dtype=int32), array([110, 291, 141, 319], dtype=int32)], [0.9104384, 0.91341376, 0.894918], [0, 0, 0]]

**************************************************

image_path /data/data_video/24/pgie/images/0_10-8-2023-17-46_17_72020.png, batch_pred: [[array([154, 136, 201, 189], dtype=int32), array([175, 290, 213, 320], dtype=int32), array([110, 291, 141, 319], dtype=int32)], [0.98416233, 0.91341376, 0.894918], [0, 0, 0]]

In the two images above, both detected two objects with a confidence score of 0.91341376, 0.894918 in the image. However, in reality, the two images above have no such object, this result is the result of a few previous images.

My sample code is saved at gitlab, please take a look

Katsuya Hyodo · Answer 11 · Fri Sep 08 2023 00:49:32 GMT+0800 (China Standard Time)

I don't understand what you are talking about.

I will not look into it unless you provide me with test images and models.

Jeff Hoang · Answer 12 · Fri Sep 08 2023 01:02:18 GMT+0800 (China Standard Time)

Sorry for that. Please allow me to send the model and data to you privately Model: yolov7, num_classes = 5, input_size 320x256 best_672.pt <https://drive.google.com/file/d/1py84NNymqSFHKb2QOFJQcXMuAJ1M1mer/view?usp=drive_web> Step to convert model: 1. From WongKinYiu/yolov7 export .pt model to onnx: python3 export.py --weights best_672.pt --simplify --grid --img-size 256 320 --dynamic-batch 2. Then use PINTO yolov7 post processing tool to get yolov7_post.onnx model 3. Convert yolov7_post.onnx model to tensorrt /usr/src/tensorrt/bin/trtexec --onnx=src_model.onnx --saveEngine=dst_model.trt --fp16 --explicitBatch --minShapes=images:1x3x256x320 --maxShapes=images:4x3x256x320 --optShapes=images:2x3x256x320 --workspace=2048 Data is attached in this email

…

On Thu, Sep 7, 2023 at 11:49 PM Katsuya Hyodo ***@***.***> wrote: I don't understand what you are talking about. I will not look into it unless you provide me with test images and models. — Reply to this email directly, view it on GitHub <#358 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMWDRMSHWFLB3WQIWFGDEBLXZH3KNANCNFSM6AAAAAA3WKOJDY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

Katsuya Hyodo · Answer 13 · Fri Sep 08 2023 08:59:05 GMT+0800 (China Standard Time)

It is no surprise that merging single-batch post-processing into a multi-batch main model will corrupt the output.