GPU mode crash - need Typesense GPU dependencies source
linuxpham opened this issue · comments
Hi admin,
When I run typesense server inside a docker. typesense server crashes. (ubuntu 20.04)
I20240308 08:59:37.412551 1346950 text_embedder.cpp:32] Loading model from disk: /data/ub-hoapha-2a9011591cbe816/vscode/multiindexing/data/typesense/models/all-MiniLM-L12-v2/model.onnx
I20240308 08:59:40.320572 1346933 raft_server.cpp:557] Term: 3, pending_queue: 0, last_index: 6, committed: 6, known_applied: 6, applying: 0, pending_writes: 0, queued_writes: 1, local_sequence: 7
I20240308 08:59:40.320668 1346981 raft_server.h:60] Peer refresh succeeded!
E20240308 08:59:47.981904 1346950 backward.hpp:4200] Stack trace (most recent call last) in thread 1346950:
E20240308 08:59:47.981997 1346950 backward.hpp:4200] #23 Object "", at 0xffffffffffffffff, in
E20240308 08:59:47.982008 1346950 backward.hpp:4200] #22 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f5cb56689ff, in
E20240308 08:59:47.982020 1346950 backward.hpp:4200] #21 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f5cb55d6b42, in
E20240308 08:59:47.982030 1346950 backward.hpp:4200] #20 Object "/data/ub-hoapha-2a9011591cbe816/.conda/envs/multiindexing/bin/typesense-server", at 0x561f51cb15c3, in execute_native_thread_routine
E20240308 08:59:47.982040 1346950 backward.hpp:4200] #19 | Source "include/threadpool.h", line 57, in operator()
E20240308 08:59:47.982049 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/future", line 1592, in _M_run
E20240308 08:59:47.982061 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/future", line 1459, in _M_set_result
E20240308 08:59:47.982072 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/future", line 412, in call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*>
E20240308 08:59:47.982084 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/mutex", line 729, in __gthread_once
E20240308 08:59:47.982093 1346950 backward.hpp:4200] Source "/usr/include/x86_64-linux-gnu/c++/10/bits/gthr-default.h", line 700, in ThreadPool [0x561f4fb50015]
E20240308 08:59:47.982102 1346950 backward.hpp:4200] #18 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f5cb55dbf67, in
E20240308 08:59:47.982111 1346950 backward.hpp:4200] #17 | Source "/usr/include/c++/10/future", line 572, in operator()
E20240308 08:59:47.982121 1346950 backward.hpp:4200] Source "/usr/include/c++/10/bits/std_function.h", line 622, in _M_do_set [0x561f4fb4ef72]
E20240308 08:59:47.982131 1346950 backward.hpp:4200] #16 | Source "/usr/include/c++/10/bits/std_function.h", line 292, in __invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run<std::_Bind<BatchedIndexer::run()::<lambda()>()>, std::allocator<int>, void, {}>::<lambda()>, void>&>
E20240308 08:59:47.982143 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/invoke.h", line 115, in __invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run<std::_Bind<BatchedIndexer::run()::<lambda()>()>, std::allocator<int>, void, {}>::<lambda()>, void>&>
E20240308 08:59:47.982154 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/invoke.h", line 60, in operator()
E20240308 08:59:47.982164 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/future", line 1397, in operator()
E20240308 08:59:47.982173 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/future", line 1456, in __invoke_r<void, std::_Bind<BatchedIndexer::run()::<lambda()>()>&>
E20240308 08:59:47.982183 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/invoke.h", line 110, in __invoke_impl<void, std::_Bind<BatchedIndexer::run()::<lambda()>()>&>
E20240308 08:59:47.982192 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/invoke.h", line 60, in operator()<>
E20240308 08:59:47.982201 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/functional", line 499, in __call<void>
E20240308 08:59:47.982210 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/functional", line 416, in __invoke<BatchedIndexer::run()::<lambda()>&>
E20240308 08:59:47.982220 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/invoke.h", line 95, in __invoke_impl<void, BatchedIndexer::run()::<lambda()>&>
E20240308 08:59:47.982230 1346950 backward.hpp:4200] Source "/usr/include/c++/10/bits/invoke.h", line 60, in _M_invoke [0x561f4fb4ceb8]
E20240308 08:59:47.982239 1346950 backward.hpp:4200] #15 Source "src/batched_indexer.cpp", line 226, in operator() [0x561f4fb4bef7]
E20240308 08:59:47.982249 1346950 backward.hpp:4200] #14 Source "src/core_api.cpp", line 200, in post_create_collection [0x561f4fc0fbf4]
E20240308 08:59:47.982257 1346950 backward.hpp:4200] #13 Source "src/collection_manager.cpp", line 1292, in create_collection [0x561f4fbf1286]
E20240308 08:59:47.982267 1346950 backward.hpp:4200] #12 Source "src/field.cpp", line 1100, in json_fields_to_fields [0x561f4fc350a3]
E20240308 08:59:47.982276 1346950 backward.hpp:4200] #11 Source "src/field.cpp", line 1142, in validate_and_init_embed_field [0x561f4fc306e5]
E20240308 08:59:47.982285 1346950 backward.hpp:4200] #10 Source "src/text_embedder_manager.cpp", line 18, in validate_and_init_model [0x561f4fd4fe23]
E20240308 08:59:47.982295 1346950 backward.hpp:4200] #9 | Source "src/text_embedder_manager.cpp", line 121, in make_shared<TextEmbedder, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982306 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/shared_ptr.h", line 878, in allocate_shared<TextEmbedder, std::allocator<TextEmbedder>, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982318 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/shared_ptr.h", line 862, in shared_ptr<std::allocator<TextEmbedder>, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982328 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/shared_ptr.h", line 408, in __shared_ptr<std::allocator<TextEmbedder>, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982339 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/shared_ptr_base.h", line 1376, in __shared_count<TextEmbedder, std::allocator<TextEmbedder>, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982349 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/shared_ptr_base.h", line 682, in _Sp_counted_ptr_inplace<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982360 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/shared_ptr_base.h", line 551, in construct<TextEmbedder, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982369 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/alloc_traits.h", line 512, in construct<TextEmbedder, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>
E20240308 08:59:47.982380 1346950 backward.hpp:4200] Source "/usr/include/c++/10/ext/new_allocator.h", line 156, in validate_and_init_local_model [0x561f4fd4f271]
E20240308 08:59:47.982390 1346950 backward.hpp:4200] #8 | Source "src/text_embedder.cpp", line 33, in make_unique<Ort::Session, Ort::Env&, char const*, Ort::SessionOptions&>
E20240308 08:59:47.982400 1346950 backward.hpp:4200] | Source "/usr/include/c++/10/bits/unique_ptr.h", line 962, in Session
E20240308 08:59:47.982409 1346950 backward.hpp:4200] Source "bazel-out/k8-fastbuild/bin/onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_inline.h", line 938, in TextEmbedder [0x561f4fd3f2d0]
E20240308 08:59:47.982419 1346950 backward.hpp:4200] #7 Object "/data/ub-hoapha-2a9011591cbe816/.conda/envs/multiindexing/bin/typesense-server", at 0x561f4fd98324, in OrtApis::CreateSession(OrtEnv const*, char const*, OrtSessionOptions const*, OrtSession**)
E20240308 08:59:47.982429 1346950 backward.hpp:4200] #6 Object "/data/ub-hoapha-2a9011591cbe816/.conda/envs/multiindexing/bin/typesense-server", at 0x561f4fd89702, in (anonymous namespace)::InitializeSession(OrtSessionOptions const*, std::unique_ptr<onnxruntime::InferenceSession, std::default_delete<onnxruntime::InferenceSession> >&, OrtPrepackedWeightsContainer*)
E20240308 08:59:47.982440 1346950 backward.hpp:4200] #5 Object "/data/ub-hoapha-2a9011591cbe816/.conda/envs/multiindexing/bin/typesense-server", at 0x561f4fdf7a21, in onnxruntime::InferenceSession::RegisterExecutionProvider(std::shared_ptr<onnxruntime::IExecutionProvider> const&)
E20240308 08:59:47.982450 1346950 backward.hpp:4200] #4 Object "/data/ub-hoapha-2a9011591cbe816/.conda/envs/multiindexing/bin/typesense-server", at 0x561f4fdf7491, in onnxruntime::ExecutionProviders::Add(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<onnxruntime::IExecutionProvider> const&)
E20240308 08:59:47.982461 1346950 backward.hpp:4200] #3 Object "/data/ub-hoapha-2a9011591cbe816/.conda/envs/multiindexing/lib/libonnxruntime_providers_cuda.so", at 0x7f5bf0f36c68, in onnxruntime::CUDAExecutionProvider::GetProviderOptions[abi:cxx11]() const
E20240308 08:59:47.982471 1346950 backward.hpp:4200] #2 Object "/data/ub-hoapha-2a9011591cbe816/.conda/envs/multiindexing/lib/libonnxruntime_providers_cuda.so", at 0x7f5bf0f4aace, in onnxruntime::CUDAExecutionProviderInfo::ToProviderOptions[abi:cxx11](onnxruntime::CUDAExecutionProviderInfo const&)
E20240308 08:59:47.982482 1346950 backward.hpp:4200] #1 Source "/home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/ostream.tcc", line 78, in _M_insert<long int> [0x7f5c50f2103d]
E20240308 08:59:47.982492 1346950 backward.hpp:4200] #0 Source "../../../../../libstdc++-v3/src/c++11/codecvt.cc", line 769, in do_unshift [0x7f5c50ee8674]
Segmentation fault (Address not mapped to object [0x20])
I20240308 08:59:49.467211 1346950 batched_indexer.cpp:392] Saving currently applying index: 6
E20240308 08:59:49.467449 1346950 typesense_server.cpp:121] Typesense 0.25.2 is terminating abruptly.
tools/typesense/start_server.sh: line 12: 1346815 Segmentation fault (core dumped) ${APP_PATH}/bin/typesense-server --api-address="0.0.0.0" --api-port=18108 --thread-pool-size=16 --data-dir=${DATA_PATH} --api-key=${TYPESENSE_API_KEY} --enable-cors
Where can we get "Typesense GPU dependencies" source (libonnxruntime_providers_cuda.so + libonnxruntime_providers_shared.so) ?
You'd only need those dependencies if you use a GPU. Links to the GPU dependencies are available in this section of the docs: https://typesense.org/docs/0.25.2/api/vector-search.html#using-a-gpu-optional
Could you also check if you can replicate this issue in 0.26.0.rc63?
0.26.0.rc63 works now. Can I get the source code of these files: libonnxruntime_providers_cuda.so, and libonnxruntime_providers_shared.so?
@linuxpham Those are compiled by ONNX Runtime, from exactly this branch