GroundingDINO的onnx转mlir报错: Assertion `op->use_empty() && "expected 'op' to have no uses"' failed.
fabro66 opened this issue · comments
以下是GroundingDINO的onnx模型转mlir的日志：
root@26d73eb7a565:/workspace/project/GroundingDINO# model_transform.py --model_name groundingdino --model_def onnx/GroundingDINOExport_16_Tokens.onnx --input_shapes [[1,3,544,960],[1,1,1,16],[1,1,1,16],[1,1,1,16],[1,1,1,16],[1,1,16,16]] --mlir mlir/groundingdino.mlir --test_input ./input/input_data.npz --test_result grounding_dino_output_data.npz
SOPHGO Toolchain v1.5.beta.0-20231107
2023/11/08 20:14:17 - INFO :
_____________________________________________________
| preprocess: |
| (x - mean) * scale |
'-------------------------------------------------------'
config Preprocess args :
resize_dims : same to net input dims
keep_aspect_ratio : False
keep_ratio_mode : letterbox
pad_value : 0
pad_type : center
--------------------------
mean : [0.0, 0.0, 0.0]
scale : [1.0, 1.0, 1.0]
--------------------------
pixel_format : bgr
channel_format : nchw
Input_shape assigned
ConstantFolding finished
skip_fuse_bn: False
Onnxsim opt finished
ConstantFolding finished
Save mlir file: mlir/groundingdino_origin.mlir
[Running]: tpuc-opt mlir/groundingdino_origin.mlir --shape-infer --canonicalize --extra-optimize -o mlir/groundingdino.mlir
tpuc-opt: /llvm-project/mlir/lib/IR/PatternMatch.cpp:305: virtual void mlir::RewriterBase::eraseOp(mlir::Operation *): Assertion `op->use_empty() && "expected 'op' to have no uses"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0. Program arguments: tpuc-opt mlir/groundingdino_origin.mlir --init --shape-infer --canonicalize --extra-optimize --deinit --mlir-print-debuginfo -o mlir/groundingdino.mlir
#0 0x000055e5ccf37157 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/workspace/install/bin/tpuc-opt+0x5f7157)
#1 0x000055e5ccf34e7e llvm::sys::RunSignalHandlers() (/workspace/install/bin/tpuc-opt+0x5f4e7e)
#2 0x000055e5ccf37ada SignalHandler(int) Signals.cpp:0:0
#3 0x00007fac0cffb520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
#4 0x00007fac0d04f9fc pthread_kill (/lib/x86_64-linux-gnu/libc.so.6+0x969fc)
#5 0x00007fac0cffb476 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x42476)
#6 0x00007fac0cfe17f3 abort (/lib/x86_64-linux-gnu/libc.so.6+0x287f3)
#7 0x00007fac0cfe171b (/lib/x86_64-linux-gnu/libc.so.6+0x2871b)
#8 0x00007fac0cff2e96 (/lib/x86_64-linux-gnu/libc.so.6+0x39e96)
#9 0x000055e5cde82c31 (/workspace/install/bin/tpuc-opt+0x1542c31)
#10 0x000055e5cdb20484 PermuteFuse::matchAndRewrite(tpu_mlir::top::PermuteOp, mlir::PatternRewriter&) const (/workspace/install/bin/tpuc-opt+0x11e0484)
#11 0x000055e5cdc726b7 void llvm::function_ref<void ()>::callback_fn<mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>)::$_2>(long) PatternApplicator.cpp:0:0
#12 0x000055e5cdc6efcf mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>) (/workspace/install/bin/tpuc-opt+0x132efcf)
#13 0x000055e5cdc3876c (anonymous namespace)::GreedyPatternRewriteDriver::processWorklist() GreedyPatternRewriteDriver.cpp:0:0
#14 0x000055e5cdc355bc mlir::applyPatternsAndFoldGreedily(mlir::Region&, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig, bool*) (/workspace/install/bin/tpuc-opt+0x12f55bc)
#15 0x000055e5cdc14bcb (anonymous namespace)::Canonicalizer::runOnOperation() Canonicalizer.cpp:0:0
#16 0x000055e5cdc9b9a4 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) (/workspace/install/bin/tpuc-opt+0x135b9a4)
#17 0x000055e5cdc9bfd1 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) (/workspace/install/bin/tpuc-opt+0x135bfd1)
#18 0x000055e5cdc9e478 mlir::PassManager::run(mlir::Operation*) (/workspace/install/bin/tpuc-opt+0x135e478)
#19 0x000055e5ccf2899b performActions(llvm::raw_ostream&, std::shared_ptr<llvm::SourceMgr> const&, mlir::MLIRContext*, mlir::MlirOptMainConfig const&) MlirOptMain.cpp:0:0
#20 0x000055e5ccf27d64 mlir::LogicalResult llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>::callback_fn<mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&)::$_2>(long, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&) MlirOptMain.cpp:0:0
#21 0x000055e5cdea2118 mlir::splitAndProcessBuffer(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>, llvm::raw_ostream&, bool, bool) (/workspace/install/bin/tpuc-opt+0x1562118)
#22 0x000055e5ccf2206a mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&) (/workspace/install/bin/tpuc-opt+0x5e206a)
#23 0x000055e5ccf22534 mlir::MlirOptMain(int, char**, llvm::StringRef, mlir::DialectRegistry&) (/workspace/install/bin/tpuc-opt+0x5e2534)
#24 0x000055e5ccf212b9 main (/workspace/install/bin/tpuc-opt+0x5e12b9)
#25 0x00007fac0cfe2d90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
#26 0x00007fac0cfe2e40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
#27 0x000055e5ccf20f15 _start (/workspace/install/bin/tpuc-opt+0x5e0f15)
Aborted (core dumped)
Traceback (most recent call last):
File "/workspace/python/tools/model_transform.py", line 273, in <module>
tool.model_transform(args.mlir, args.add_postprocess)
File "/workspace/python/tools/model_transform.py", line 51, in model_transform
mlir_opt_for_top(mlir_origin, self.mlir_file, add_postprocess)
File "/workspace/python/utils/mlir_shell.py", line 60, in mlir_opt_for_top
_os_system(cmd)
File "/workspace/python/utils/mlir_shell.py", line 50, in _os_system
raise RuntimeError("[!Error]: {}".format(cmd_str))
RuntimeError: [!Error]: tpuc-opt mlir/groundingdino_origin.mlir --shape-infer --canonicalize --extra-optimize -o mlir/groundingdino.mlir
请问这是什么原因造成的呢？如果我提前采用onnxsim对模型简化，就不会在这个阶段报错，但在下一步对比mlir和onnx相似度的euclidean_similarity指标过不了。如果不考虑euclidean_similarity误差，直接转bm1684x的fp32模型，推理出来的结果也不对。
我也遇到了同样的问题，无法解决
请问提前用onnxsim对模型简化这一步具体是怎么操作呢