alibaba / BladeDISC

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

coredump when set tf_enable_tao be true

houjincheng1992 opened this issue · comments

commented

tensorflow version is 1.15.5, cuda version cuda-10.1

when set TF_ENABLE_TAO true, coredump as following:

2023-07-17 21:34:47.554155: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:288] Restoring SavedModel bundle.
*** Aborted at 1689600887 (unix time) try "date -d @1689600887" if you are using GNU date ***
PC: @ 0x7f62463f39d4 tensorflow::tao::GetGlobalJitLevelForGraph()
*** SIGSEGV (@0x400000011) received by PID 288 (TID 0x7f6031358700) from PID 17; stack trace: ***
@ 0x7f629ffe0929 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f62a06a25f0 (unknown)
@ 0x7f62463f39d4 tensorflow::tao::GetGlobalJitLevelForGraph()
@ 0x7f624638fafa tensorflow::tao::(anonymous namespace)::MarkForCompilation()
@ 0x7f6246392fb0 tensorflow::tao::TaoMarkForCompilationPass::Run()
@ 0x7f624639403c tensorflow::tao::TaoOptimizationPass::Run()
@ 0x7f62b0ffb196 tensorflow::OptimizationPassRegistry::RunGrouping()
@ 0x7f62aeae25e0 tensorflow::GraphExecutionState::BuildGraph()
@ 0x7f62aea245da tensorflow::DirectSession::CreateGraphs()
@ 0x7f62aea25fc7 tensorflow::DirectSession::CreateExecutors()
@ 0x7f62aea29689 tensorflow::DirectSession::MakeCallable()
@ 0x7f62a7a3a95d tensorflow::(anonymous namespace)::RunOnce()
@ 0x7f62a7a3b56d tensorflow::(anonymous namespace)::RunRestore()
@ 0x7f62a7a3cdce tensorflow::(anonymous namespace)::LoadSavedModelInternal()
@ 0x7f62a7a3d8e4 tensorflow::LoadSavedModel()
@ 0x7f62a79eeffb tensorflow::serving::LoadSessionBundleOrSavedModelBundle()
@ 0x7f62a78d5bf5 tensorflow::serving::SavedModelBundleV2Factory::InternalCreateSavedModelBundleV2()
@ 0x7f62a78d6781 tensorflow::serving::SavedModelBundleV2Factory::CreateSavedModelBundleV2()
@ 0x7f62a78d1887 ZNSt17_Function_handlerIFN10tensorflow6StatusEPSt10unique_ptrINS0_18SavedModelBundleV2ESt14default_deleteIS3_EEEZNKS0_7serving31SavedModelBundleV2SourceAdapter18GetServableCreatorESt10shared_ptrINS9_25SavedModelBundleV2FactoryEERKSsEUlS7_E_E9_M_invokeERKSt9_Any_dataOS7
@ 0x7f62a78cf65e tensorflow::serving::SimpleLoader<>::Load()
@ 0x7f62a78b67b9 _ZNSt17_Function_handlerIFN10tensorflow6StatusEvEZNS0_7serving13LoaderHarness4LoadEvEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x7f62a78b90ea tensorflow::serving::Retry()
@ 0x7f62a78b7ec6 tensorflow::serving::LoaderHarness::Load()
@ 0x7f62a78b3603 tensorflow::serving::BasicManager::ExecuteLoad()
@ 0x7f62a78b3b8c tensorflow::serving::BasicManager::ExecuteLoadOrUnload()
@ 0x7f62a78b56a2 tensorflow::serving::BasicManager::HandleLoadOrUnloadRequest()
@ 0x7f62a78b5892 _ZNSt17_Function_handlerIFvvEZN10tensorflow7serving12BasicManager20LoadOrUnloadServableERKNS3_19LoadOrUnloadRequestESt8functionIFvRKNS1_6StatusEEEEUlvE0_E9_M_invokeERKSt9_Any_data
@ 0x7f62b177be0c Eigen::ThreadPoolTempl<>::WorkerLoop()
@ 0x7f62b1778d98 _ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x7f629e1c7b9f execute_native_thread_routine
@ 0x7f62a069ae65 start_thread
@ 0x7f629dc2988d __clone