jina-ai / clip-as-service

🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP

Home Page:https://clip-as-service.jina.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

trtexec throws error related to constant_folding(trt v8601)

thunder95 opened this issue · comments

FAILED TensorRT.trtexec [TensorRT v8601]
[optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/model/layers.31/self_attn/Slice_6_output_0[Constant].../lm_head/MatMul]}.)

Tried to convert a llm to trt model, used pytorch 2.2.0 cpu, with the following parameters:
org_dummy_input = make_train_dummy_input(128)
output = torch.onnx.export(model,
org_dummy_input,
"baichuan2.onnx",
verbose=False,
do_constant_folding=False,
opset_version=14,
input_names=['input_ids', 'attention_mask'],
output_names=['lm_head'],
dynamic_axes={"input_ids": {0: "batch_size", 1: "sequence_length"},
"attention_mask": {0: "batch_size", 1: "sequence_length"},
"lm_head": {0: "batch_size"}})

This model is from https://github.com/baichuan-inc/Baichuan2.
And this error is quite similar to this issue: NVIDIA/TensorRT#2124.
Please help me, thanks!