Giters

envoyproxy / envoy-wasm

*ATTENTION!: The content of this repo is merged into https://github.com/envoyproxy/envoy and future development is happening there.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Envoy crashes on exit

kyessenov opened this issue · comments

commented

Segfault detected when calling quitquitquit:

[2020-03-26 13:46:09.645][18719][info][main] [external/envoy/source/server/server.cc:627] exiting
[2020-03-26 13:46:09.657][18719][critical][assert] [external/envoy/source/common/stats/thread_local_store.cc:36] assert failure: scopes_.empty().
[2020-03-26 13:46:09.657][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3ea0000491f
[2020-03-26 13:46:09.657][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-03-26 13:46:09.657][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: c785a47b22ca50631ab0076328fd8a5f95707ce0/1.14.0-dev/Clean/DEBUG/BoringSSL
[2020-03-26 13:46:09.694][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x55d4c1be299c]
[2020-03-26 13:46:09.694][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __restore_rt [0x7f8a4ba42890]
[2020-03-26 13:46:09.729][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: Envoy::Stats::ThreadLocalStoreImpl::~ThreadLocalStoreImpl() [0x55d4c11cf6d9]
[2020-03-26 13:46:09.763][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: std::default_delete<>::operator()() [0x55d4bed465bf]
[2020-03-26 13:46:09.798][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::unique_ptr<>::~unique_ptr() [0x55d4bed45e70]
[2020-03-26 13:46:09.832][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: Envoy::MainCommonBase::~MainCommonBase() [0x55d4bed45a0b]
[2020-03-26 13:46:09.866][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::MainCommon::~MainCommon() [0x55d4bed45935]
[2020-03-26 13:46:09.901][18719][critical][backtrace] [bazel-out/k8-fastbuild/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: std::default_delete<>::operator()() [0x55d4bed457eb]

@jplevyak

commented

This is on master of istio/proxy, in one of the integration tests.

commented

Is this from a Stackdriver filter test? Wondering could it because of the recent change flushing log entry in onDone.

commented

Yes, I think so.

commented

@jplevyak We think this is because envoy exits before the module finishes its work. This has a risk of losing telemetry on restart. Can we make the host wait until the module is done before exiting?

commented

We talked to Matt about this, and yes we can do it. It will require some changes to Envoy proper.