Crash under heavy data/control load
kyessenov opened this issue · comments
At 3kqps and control plane updating vm_id for metadata_exchange v8 raw wasm every 2s using ECDS, I'm seeing a crash about 90 minutes in:
[2020-08-11 15:51:26.435][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x55c100000000
[2020-08-11 15:51:26.435][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-11 15:51:26.435][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: e275bca85d3412681b18275274d61c55573db731/1.15.0/Modified/DEBUG/BoringSSL
[2020-08-11 15:51:26.502][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x55c10bc7433c]
[2020-08-11 15:51:26.502][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7f55d3e4c8a0]
[2020-08-11 15:51:26.560][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: proxy_wasm::(anonymous namespace)::V8::getModuleFunctionImpl<>()::{lambda()#1}::operator()() [0x55c109f73ca5]
[2020-08-11 15:51:26.617][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: std::_Function_handler<>::_M_invoke() [0x55c109f73a62]
[2020-08-11 15:51:26.675][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::function<>::operator()() [0x55c109f4eda0]
[2020-08-11 15:51:26.732][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: proxy_wasm::ContextBase::onLog() [0x55c109f4b0bc]
[2020-08-11 15:51:26.790][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Extensions::Common::Wasm::Context::log() [0x55c109be57d3]
[2020-08-11 15:51:26.847][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x55c10b75950d]
[2020-08-11 15:51:26.905][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x55c10b759e20]
[2020-08-11 15:51:26.962][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x55c10b759f19]
[2020-08-11 15:51:27.019][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: std::default_delete<>::operator()() [0x55c108c907df]
[2020-08-11 15:51:27.077][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: std::unique_ptr<>::reset() [0x55c109c2a5c6]
[2020-08-11 15:51:27.134][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::Event::DispatcherImpl::clearDeferredDeleteList() [0x55c10b179bb0]
[2020-08-11 15:51:27.192][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Event::DispatcherImpl::DispatcherImpl()::$_0::operator()() [0x55c10b17c0f9]
[2020-08-11 15:51:27.250][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: std::_Function_handler<>::_M_invoke() [0x55c10b17bfad]
[2020-08-11 15:51:27.307][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: std::function<>::operator()() [0x55c108704bce]
[2020-08-11 15:51:27.365][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::operator()() [0x55c10b1b0cb5]
[2020-08-11 15:51:27.422][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #17: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::__invoke() [0x55c10b1b0c79]
[2020-08-11 15:51:27.480][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #18: event_process_active_single_queue [0x55c10bb79be9]
[2020-08-11 15:51:27.537][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #19: event_process_active [0x55c10bb7415b]
[2020-08-11 15:51:27.595][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #20: event_base_loop [0x55c10bb72fce]
[2020-08-11 15:51:27.652][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #21: Envoy::Event::LibeventScheduler::run() [0x55c10b1aeaac]
[2020-08-11 15:51:27.709][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #22: Envoy::Event::DispatcherImpl::run() [0x55c10b17bb28]
[2020-08-11 15:51:27.766][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #23: Envoy::Server::WorkerImpl::threadRoutine() [0x55c10b158219]
[2020-08-11 15:51:27.823][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #24: Envoy::Server::WorkerImpl::start()::$_4::operator()() [0x55c10b15994f]
[2020-08-11 15:51:27.880][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #25: std::_Function_handler<>::_M_invoke() [0x55c10b1597fd]
[2020-08-11 15:51:27.880][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: std::function<>::operator()() [0x55c108704bce]
[2020-08-11 15:51:27.937][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #27: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x55c10bcd2a70]
[2020-08-11 15:51:27.994][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #28: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x55c10bcd2a45]
[2020-08-11 15:51:27.994][22909][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #29: start_thread [0x7f55d3e416db]
Segmentation fault
This does not appear to be OOM since I was monitoring the physical bytes (hovering around 100MB).
At 1qps config reload, happens much sooner:
[2020-08-11 21:13:00.416][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x560500000000
[2020-08-11 21:13:00.416][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-11 21:13:00.416][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: e275bca85d3412681b18275274d61c55573db731/1.15.0/Modified/DEBUG/BoringSSL
[2020-08-11 21:13:00.478][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x560523a9f33c]
[2020-08-11 21:13:00.479][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7f19533f68a0]
[2020-08-11 21:13:00.536][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: proxy_wasm::(anonymous namespace)::V8::getModuleFunctionImpl<>()::{lambda()#1}::operator()() [0x560521d9eca5]
[2020-08-11 21:13:00.594][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: std::_Function_handler<>::_M_invoke() [0x560521d9ea62]
[2020-08-11 21:13:00.652][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::function<>::operator()() [0x560521d79da0]
[2020-08-11 21:13:00.709][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: proxy_wasm::ContextBase::onLog() [0x560521d760bc]
[2020-08-11 21:13:00.767][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Extensions::Common::Wasm::Context::log() [0x560521a107d3]
[2020-08-11 21:13:00.825][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x56052358450d]
[2020-08-11 21:13:00.883][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x560523584e20]
[2020-08-11 21:13:00.941][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() [0x560523584f19]
[2020-08-11 21:13:00.999][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: std::default_delete<>::operator()() [0x560520abb7df]
[2020-08-11 21:13:01.056][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: std::unique_ptr<>::reset() [0x560521a555c6]
[2020-08-11 21:13:01.114][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::Event::DispatcherImpl::clearDeferredDeleteList() [0x560522fa4bb0]
[2020-08-11 21:13:01.173][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Event::DispatcherImpl::DispatcherImpl()::$_0::operator()() [0x560522fa70f9]
[2020-08-11 21:13:01.232][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: std::_Function_handler<>::_M_invoke() [0x560522fa6fad]
[2020-08-11 21:13:01.290][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: std::function<>::operator()() [0x56052052fbce]
[2020-08-11 21:13:01.347][26263][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::operator()() [0x560522fdbcb5]
It looks that it's trying to the log request after the VM is already deleted. Is it possible that the removed (i.e. previous) ExtensionConfig removes the VM, even though it's still handling requests, @jplevyak?
Could you try with ASan build?
==29804==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d0003a5881 at pc 0x55f0c86441db bp 0x7f5d0fecee90 sp 0x7f5d0fecee88
READ of size 1 at 0x61d0003a5881 thread T6
#0 0x55f0c86441da in proxy_wasm::WasmBase::isFailed() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x57ab01da)
#1 0x55f0c94e0f93 in proxy_wasm::ContextBase::isFailed() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5894cf93)
#2 0x55f0c94f53f0 in proxy_wasm::ContextBase::onLog() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x589613f0)
#3 0x55f0c84f91c1 in Envoy::Extensions::Common::Wasm::Context::log(Envoy::Http::RequestHeaderMap const*, Envoy::Http::ResponseHeaderMap const*, Envoy::Http::ResponseTrailerMap const*, Envoy::StreamInfo::StreamInfo const&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x579651c1)
#4 0x55f0cf524989 in Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5e990989)
#5 0x55f0cf525d05 in Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5e991d05)
#6 0x55f0cf525e8e in Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5e991e8e)
#7 0x55f0c43bcef5 in std::default_deleteEnvoy::Event::DeferredDeletable::operator()(Envoy::Event::DeferredDeletable*) const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x53828ef5)
#8 0x55f0c864231f in std::unique_ptr<Envoy::Event::DeferredDeletable, std::default_deleteEnvoy::Event::DeferredDeletable >::reset(Envoy::Event::DeferredDeletable*) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x57aae31f)
#9 0x55f0cd88ac0e in Envoy::Event::DispatcherImpl::clearDeferredDeleteList() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5ccf6c0e)
#10 0x55f0cd8937c7 in Envoy::Event::DispatcherImpl::DispatcherImpl(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::unique_ptr<Envoy::Buffer::WatermarkFactory, std::default_deleteEnvoy::Buffer::WatermarkFactory >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0::operator()() const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5ccff7c7)
#11 0x55f0cd8932ae in std::_Function_handler<void (), Envoy::Event::DispatcherImpl::DispatcherImpl(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::unique_ptr<Envoy::Buffer::WatermarkFactory, std::default_deleteEnvoy::Buffer::WatermarkFactory >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0>::_M_invoke(std::_Any_data const&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5ccff2ae)
#12 0x55f0c2a973c9 in std::function<void ()>::operator()() const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x51f033c9)
#13 0x55f0cd9781fc in Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl(Envoy::CSmartPtr<event_base, &(event_base_free)>&, std::function<void ()>)::$_0::operator()(int, short, void*) const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5cde41fc)
#14 0x55f0cd978148 in Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl(Envoy::CSmartPtr<event_base, &(event_base_free)>&, std::function<void ()>)::$_0::__invoke(int, short, void*) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5cde4148)
#15 0x55f0d0832f02 in event_process_active_single_queue (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5fc9ef02)
#16 0x55f0d080b8e9 in event_process_active (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5fc778e9)
#17 0x55f0d080354c in event_base_loop (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5fc6f54c)
#18 0x55f0cd96e003 in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5cdda003)
#19 0x55f0cd89219a in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5ccfe19a)
#20 0x55f0cd7f2f1f in Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5cc5ef1f)
#21 0x55f0cd7f8b52 in Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4::operator()() const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5cc64b52)
#22 0x55f0cd7f861e in std::_Function_handler<void (), Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4>::_M_invoke(std::_Any_data const&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5cc6461e)
#23 0x55f0c2a973c9 in std::function<void ()>::operator()() const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x51f033c9)
#24 0x55f0d0e60303 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>, absl::optionalEnvoy::Thread::Options const&)::'lambda'(void*)::operator()(void*) const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x602cc303)
#25 0x55f0d0e60264 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function<void ()>, absl::optionalEnvoy::Thread::Options const&)::'lambda'(void*)::__invoke(void*) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x602cc264)
#26 0x7f5d181316da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#27 0x7f5d178b9a3e in clone /build/glibc-2ORdQG/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
previously allocated by thread T6 here:
#0 0x55f0c25209d3 in __interceptor_malloc (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x5198c9d3)
#1 0x7f5d17e34297 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x93297)
#2 0x55f0c302bcb5 in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<Envoy::Extensions::Common::Wasm::Istio::IstioWasm, std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, (__gnu_cxx::_Lock_policy)2> > >::allocate(std::allocator<std::_Sp_counted_ptr_inplace<Envoy::Extensions::Common::Wasm::Istio::IstioWasm, std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, (__gnu_cxx::_Lock_policy)2> >&, unsigned long) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x52497cb5)
#3 0x55f0c302af9e in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<Envoy::Extensions::Common::Wasm::Istio::IstioWasm, std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, (__gnu_cxx::_Lock_policy)2> > > std::__allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<Envoy::Extensions::Common::Wasm::Istio::IstioWasm, std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, (__gnu_cxx::_Lock_policy)2> > >(std::allocator<std::_Sp_counted_ptr_inplace<Envoy::Extensions::Common::Wasm::Istio::IstioWasm, std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, (__gnu_cxx::_Lock_policy)2> >&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x52496f9e)
#4 0x55f0c304ec3b in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<Envoy::Extensions::Common::Wasm::Istio::IstioWasm, std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, std::shared_ptrEnvoy::Extensions::Common::Wasm::WasmHandle const&, Envoy::Event::Dispatcher&>(Envoy::Extensions::Common::Wasm::Istio::IstioWasm*&, std::_Sp_alloc_shared_tag<std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm >, std::shared_ptrEnvoy::Extensions::Common::Wasm::WasmHandle const&, Envoy::Event::Dispatcher&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x524bac3b)
#5 0x55f0c304e8ff in std::__shared_ptr<Envoy::Extensions::Common::Wasm::Istio::IstioWasm, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, std::shared_ptrEnvoy::Extensions::Common::Wasm::WasmHandle const&, Envoy::Event::Dispatcher&>(std::_Sp_alloc_shared_tag<std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm >, std::shared_ptrEnvoy::Extensions::Common::Wasm::WasmHandle const&, Envoy::Event::Dispatcher&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x524ba8ff)
#6 0x55f0c304e520 in std::shared_ptrEnvoy::Extensions::Common::Wasm::Istio::IstioWasm::shared_ptr<std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm, std::shared_ptrEnvoy::Extensions::Common::Wasm::WasmHandle const&, Envoy::Event::Dispatcher&>(std::_Sp_alloc_shared_tag<std::allocatorEnvoy::Extensions::Common::Wasm::Istio::IstioWasm >, std::shared_ptrEnvoy::Extensions::Common::Wasm::WasmHandle const&, Envoy::Event::Dispatcher&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x524ba520)
If I had to guess, this looks like a lifetime issue. The ActiveStream is being destroyed by a deferred delete and it is occurring after the filter chain along with the Wasm VM has been destroyed. It looks like the UpstreamRequest destructor is calling the upstream_logs_ ->log() call which in turn is calling into the (already destroyed) VM.
I can work around this, but it seems like a larger problem.
Why is upstream log hooked into the HTTP filter here? That does sound like a problem if upstreams can send log calls to downstreams.
No longer happens. Seems like log() fix worked.