Discussion about scrfd running tensorrt

[Khanh Tran]

Hi. This's not related to this codebase yet but we may have the same concern so I post it here. I'm using onnx with tensorrt backend trying to do inference with model scrfd_10g_bnkps. The problem comes when output from tensorrt is different with onnx backend. In addition, anchor_centers and bbox_preds do not have matched shapes after predicting with tensorrt. Detail below:

onnx_runtime:
anchor_centers: (12800, 2)
bbox_preds: (12800, 4)
anchor_centers: (3200, 2)
bbox_preds: (3200, 4)
anchor_centers: (800, 2)
bbox_preds: (800, 4)

tensorrt:
anchor_centers: (12800, 2)
bbox_preds: (3200, 1)
anchor_centers: (3200, 2)
bbox_preds: (3200, 4)
anchor_centers: (800, 2)
bbox_preds: (3200, 10)

I hope you can take your time looking into this.

[Khanh Tran]

import onnx
import onnx_tensorrt.backend as backend
import numpy as np
import cv2
model = onnx.load(".insightface/models/antelope/scrfd_10g_bnkps.onnx")

engine = backend.prepare(model, device='CUDA:0')
img = cv2.imread("test.jpg")
img = cv2.resize(img, (640, 640))
input_size = tuple(img.shape[0:2][::-1])
blob = cv2.dnn.blobFromImage(img, 1.0/128, input_size, (127.5, 127.5, 127.5), swapRB=True)
net_outs = engine.run(blob)

input_height = blob.shape[2]
input_width = blob.shape[3]
fmc = 3
center_cache = {}
use_kps = True
_feat_stride_fpn = [8, 16, 32]
_num_anchors = 2
threshold = 0.5
scores_list = []
bboxes_list = []
kpss_list = []
for idx, stride in enumerate(_feat_stride_fpn):
    scores = net_outs[idx]
    bbox_preds = net_outs[idx+fmc]
    bbox_preds = bbox_preds * stride
    if use_kps:
        kps_preds = net_outs[idx+fmc*2] * stride
    height = input_height // stride
    width = input_width // stride
    K = height * width
    key = (height, width, stride)
    if key in center_cache:
        anchor_centers = center_cache[key]
    else:
        anchor_centers = np.stack(np.mgrid[:height, :width][::-1], axis=-1).astype(np.float32)
        anchor_centers = (anchor_centers * stride).reshape( (-1, 2) )
        if _num_anchors>1:
            anchor_centers = np.stack([anchor_centers]*_num_anchors, axis=1).reshape( (-1,2) )
        if len(center_cache)<100:
            center_cache[key] = anchor_centers

    pos_inds = np.where(scores>=threshold)[0]
    print("anchor_centers: ", anchor_centers.shape)
    print("bbox_preds: ", bbox_preds.shape)

[SthPhoenix]

I have added support for scrfd in this project, though I haven't noticed any difference in ONNX and TRT outputs yet.

[Khanh Tran]

@SthPhoenix As I can see, scrfd model in your codebase is done inference using onnxruntime, not tensorrt

[SthPhoenix]

@SthPhoenix As I can see, scrfd model in your codebase is done inference using onnxruntime, not tensorrt

It supports both backends.

[dexception]

As far as i can see the output is similar but the sequence of the output is different.

[SthPhoenix]

Yes, for some reason this problem really exist, that's why right output order is passed to onnxrt engine and is used for reformatting of TRT outputs.