When batch>2 or dynamic, can not conver onnx to trt

Question

When batch>2 or dynamic, can not conver onnx to trt

CHR1122 opened this issue 5 months ago · comments

It seems to me like the max detections parameter in the line final_dets = batched_dets.new_zeros((b, self.max_det, i)) is causing the issue, as this is set to 100 statically regardless of batch size, causing the reshape to [.., 100, ...]. You can modify the forward pass of the DeepstreamOutput class in export_yoloV8_seg.py to multiply the max detections with the batch size, specifically, if you add to the beginning of the forward pass:
dynamic_max_det = x[0].shape[0] * self.max_det
And then modify line 98 to:
final_dets = batched_dets.new_zeros((b, dynamic_max_det, i))
While this works and the exported ONNX model can be used to successfully build an engine with a dynamic batch size, this does decimate my throughput when used on a single UDP stream input for some reason. Going from ~15FPS on batch size = 1 with an AGX Xavier on Jetpack 5.0.2 to ~3FPS on batch size = 2 and 4. Might be an issue with my modification or perhaps an issue with how I'm applying batching in this context. Would be curious to see if this causes similar issues for others.

I've tried this and it still doesn't work.

pt2onnx run:
python export_yoloV8_seg.py -w yolov8s-seg.pt --dynamic --simplify
and onnx2trt run:
trtexec --onnx=./onnx_models/yolov8s-seg.onnx --saveEngine=./engine_models/yolov8s-seg.engine --fp16 --minShapes=input:1x3x640x640 --optShapes=input:4x3x640x640 --maxShapes=input:4x3x640x640
The error message is still reported, and the error message is as follows:

[02/20/2024-10:50:42] [W] [TRT] /1/Reshape_12: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 1 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:43] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +10, now: CPU 690, GPU 948 (MiB)
[02/20/2024-10:50:43] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +11, GPU +9, now: CPU 701, GPU 957 (MiB)
[02/20/2024-10:50:43] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[02/20/2024-10:50:43] [E] Error[4]: [shapeCompiler.cpp::evaluateShapeChecks::1180] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: IShuffleLayer /1/Reshape_12: reshaping failed for tensor: /1/Expand_1_output_0 Reshape would change volume.)
[02/20/2024-10:50:43] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[02/20/2024-10:50:43] [E] Engine could not be created from network
[02/20/2024-10:50:43] [E] Building engine failed
[02/20/2024-10:50:43] [E] Failed to create engine from model or file.
[02/20/2024-10:50:43] [E] Engine set up failed```