high inference times for initial images

[Saimouli Katragadda]

It takes a long time to do inference for the first few images. Progressively the inference time reduces. Does engine generation optimize according to the device? Any cache/changes needed to make the inference time constant for any number of iterations?

000273.png
103ms preprocess
419ms inference

000237.png
2ms preprocess
14ms inference

000103.png
2ms preprocess
14ms inference

[Vijaysrinivas Rajagopal]

I believe this is just an unfortunate limitation of GPU kernel initialization. There is not much you can do other than maybe running dummy inputs for however many slow inferences there are before going into your real data.