how to apply post process on openvino inference?

[MinGiSa]

💡 Your Question

========================================

source = "Flat_00002.jpg"
PRETRAINED_PATH = "checkPoint\ckpt_best.pth"

image = cv2.imread(source)
imageResize = cv2.resize(image, (640, 640))
imageTransPose = np.transpose(imageResize, (2, 0, 1))
imageNormalize = (imageTransPose / 255).astype(np.float32)
imageExpandDim = np.expand_dims(imageNormalize, axis=0)

compiled_model = ov.compile_model("openVino.xml")

result = compiled_model(imageExpandDim)

print(result)

==========================================

When I try to infer with the above code, I get the following result:

{<ConstOutput: names[] shape[?,3..,4] type: f32>: array([[[-4.9871521e+00, -1.1022096e+00, 4.0015339e+01, 2.5291683e+01],
[-4.7160664e+00, 3.2692695e-01, 6.1891304e+01, 9.2061386e+00],
[-7.2267246e+00, 4.9339676e-01, 6.8342850e+01, 1.1670666e+01],
...,
[ 3.6447223e+02, 4.3930225e+02, 7.4624023e+02, 8.2782220e+02],
[ 3.9647223e+02, 4.3930225e+02, 7.7824023e+02, 8.2782220e+02],
[ 4.2847223e+02, 4.3930225e+02, 8.1024023e+02, 8.2782220e+02]]],
dtype=float32), <ConstOutput: names[] shape[?,?,4] type: f32>: array([[[0.007194 , 0.00572572, 0.01053155, 0.01321735],
[0.00585808, 0.00508891, 0.01034159, 0.00883044],
[0.00418563, 0.00303123, 0.0056036 , 0.00600452],
...,
[0.00196522, 0.00195 , 0.00201701, 0.00199825],
[0.00195173, 0.00193724, 0.00199909, 0.00198264],
[0.00183082, 0.00181984, 0.00186021, 0.00185179]]], dtype=float32)}

This is the kind of result I get.
Given 2 tensors of shapes (1, 8400, 4), (1, 8400, 4), these tensors correspond to the box coords (4), and classes probability (4 classes).

1 . I think I should apply NMS to get the correct result. How can I apply post-processing to it?
can you help me to show example code?
i tried to do it.

======================================================================
postprocess = PPYoloEPostPredictionCallback(score_threshold=0.5, nms_threshold=0.4, nms_top_k=200, max_predictions=100, multi_label_per_box=False)(result)

and i get.

ValueError: Unsupported output format: {<ConstOutput: names[] shape[?,3..,4] type: f32>: array([[[-4.9871521e+00, -1.1022096e+00, 4.0015339e+01, 2.5291683e+01],
[-4.7160664e+00, 3.2692695e-01, 6.1891304e+01, 9.2061386e+00],
[-7.2267246e+00, 4.9339676e-01, 6.8342850e+01, 1.1670666e+01],
...,
[ 3.6447223e+02, 4.3930225e+02, 7.4624023e+02, 8.2782220e+02],
[ 3.9647223e+02, 4.3930225e+02, 7.7824023e+02, 8.2782220e+02],
[ 4.2847223e+02, 4.3930225e+02, 8.1024023e+02, 8.2782220e+02]]],
dtype=float32), <ConstOutput: names[] shape[?,?,4] type: f32>: array([[[0.007194 , 0.00572572, 0.01053155, 0.01321735],
[0.00585808, 0.00508891, 0.01034159, 0.00883044],
[0.00418563, 0.00303123, 0.0056036 , 0.00600452],
...,
[0.00196522, 0.00195 , 0.00201701, 0.00199825],
[0.00195173, 0.00193724, 0.00199909, 0.00198264],
[0.00183082, 0.00181984, 0.00186021, 0.00185179]]], dtype=float32)}

=======================================================================

2. and Is it possible to convert to int8 with OpenVINO?, I have torch, onnx PTQ, onnx QAT model files.

Versions

PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.0 | packaged by conda-forge | (default, Nov 10 2021, 13:20:59) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22631-SP0
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070 with Max-Q Design
Nvidia driver version: 522.06
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2592
DeviceID=CPU0
Family=198
L2CacheSize=1536
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2592
Name=Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.23.0
[pip3] onnx==1.15.0
[pip3] onnxruntime==1.15.0
[pip3] onnxsim==0.4.36
[pip3] torch==2.0.1+cu117
[pip3] torchaudio==2.0.2+cu117
[pip3] torchinfo==1.8.0
[pip3] torchmetrics==0.8.0
[pip3] torchvision==0.15.2+cu117
[conda] numpy 1.23.0 pypi_0 pypi
[conda] torch 2.0.1+cu117 pypi_0 pypi
[conda] torchaudio 2.0.2+cu117 pypi_0 pypi
[conda] torchinfo 1.8.0 pypi_0 pypi
[conda] torchmetrics 0.8.0 pypi_0 pypi
[conda] torchvision 0.15.2+cu117 pypi_0 pypi

[Eugene Khvedchenya]

You can export model with postprocessing to ONNX file and from there you can convert it to OpenVINO. This way the model will have postprocessing in the graph and you can avoid doing manual postprocessing.

[MinGiSa]

have postprocessing in the graph and you can avoid doing manual postprocessing.

thank you for your answer!