trt batch infer results error

[philoxmyu]

Use trt backend, max_batch_size 2 for test.
When I perform inference with a single image, the return result is correct, but if two different images are input together, the second image will have wrong detections; but if the two images are same, the results are correct

What could be the reason for this? thanks

some errors for example, use scrfd_500m_bnkps_640_640_batch2.plan

1. when use
   [
   "test_images/Stallone.jpg",
   "test_images/mask.jpg"
   ]

2. when use
   [
   "test_images/mask.jpg",
   "test_images/lumia.jpg",
   ]

[SthPhoenix]

Hi! that's interesting, could you please run the same test with yolov5s-face model to narrow down the issue?

[SthPhoenix]

Should be working as expected now ) Absolutely dumb mistake with array offsets )

[philoxmyu]

Thanks for replying so quickly.
In addition to this offset problem, there may be another problem that needs to be changed.
When batch input for test, I found that there is a problem with the first image. After debug, self.score_list/bbox_list/kpss_list will be overwritten by the following image, resulting in an error in the previous image.
After reset, the results is ok.

[SthPhoenix]

Actually those arrays aren't reset intentionally, I found that reallocating them might noticably impact performance, so I have allocated memory during initialization and then just assigning new values.

What kind of errors you have with first image? I'll investigate it more thoroughly.

[philoxmyu]

If not reset, because different images share these arrays, when the offset starts from 0 during the batch cycle, the previous detection data will be overwritten by the latter

for two images, batch=2

[
"test_images/mask.jpg",
"test_images/lumia.jpg",
]

errors in first image:

[SthPhoenix]

Yes, I can reproduce this behavior too. Array slices are passed to output by reference instead of value, I'll fix it shortly.
Thanks for pointing out the issue!

[philoxmyu]

By the way, ask a question. When using batch inference (eg: scrfd_500m_bnkps.onnx) under gpu with onnxruntime-gpu, the batch inference time(only the inference time, not include pre and post process) is basically not improved. Have you encountered it? The model is also running on the gpu.
thanks

[SthPhoenix]

By the way, ask a question. When using batch inference (eg: scrfd_500m_bnkps.onnx) under gpu with onnxruntime-gpu, the batch inference time(only the inference time, not include pre and post process) is basically not improved. Have you encountered it? The model is also running on the gpu. thanks

I haven't added batch inference for onnxruntime, so it should be processing images one by one.

I remember there were some issues with batch inference enabled and using onnxruntime on CPU, though I can't recall if there were any issues on GPU.

[philoxmyu]

okay, thanks for your reply!