Giters

zeek / broker

Zeek's Messaging Library

Home Page:https://docs.zeek.org/projects/broker

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

caf::actor_control_block::enqueue segfault when running supervisor.config-bare-mode test

awelzel opened this issue · comments

commented

Segfault has been observed in a CI ASAN build running the supervisor.config-bare-mode/config-bare-mode.zeek and a similar one was reported by @initconf on Slack in a production environment.

Reproducer Environment:

  • 6 CPU VM running Ubuntu 22.04, zeek version 6.0.0-dev.636-debug, putting the system under load using stress.

Build script:

#!/bin/bash
...
export CFLAGS="-g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
export CXXFLAGS="-g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
export LDFLAGS="-fuse-ld=lld"

export ZEEK_TAILORED_UB_CHECKS=1

./configure \
    --ccache \
    --generator=Ninja \
    --build-dir=./build \
    --prefix=$PREFIX \
    --build-type=debug \
    --disable-broker-tests \
    --disable-btest \
    --disable-btest-pcaps \
    --disable-zeek-client \
    --disable-zeekctl \
    --sanitizers=address

awelzel@ubuntu-01:~/zeek/testing/btest$ stress -c 6 -i 1 -d 1 -m 6 &
[1] 108863
awelzel@ubuntu-01:~/zeek/testing/btest$ stress: info: [108863] dispatching hogs: 6 cpu, 1 io, 6 vm, 1 hdd

awelzel@ubuntu-01:~/zeek/testing/btest$ C=0; while ../../auxil/btest/btest -d supervisor/config-bare-mode.zeek; do C=$(( C + 1 )); echo "good $C" ; done
[  0%] supervisor.config-bare-mode ... failed
  % 'btest-bg-wait 30' failed unexpectedly (exit code 1)
  % cat .stderr
  <<< [108938] zeek -j -b /home/awelzel/zeek/testing/btest/.tmp/supervisor.config-bare-mode/config-bare-mode.zeek
  received termination signal
  AddressSanitizer:DEADLYSIGNAL
  =================================================================
  ==108940==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000040 (pc 0x7f39154a37a8 bp 0x7f390dfe6f60 sp 0x7f390dfe6e90 T4)
  ==108940==The signal is caused by a READ memory access.
  ==108940==Hint: address points to the zero page.
      #0 0x7f39154a37a8 in caf::actor_control_block::enqueue(caf::intrusive_ptr<caf::actor_control_block>, caf::message_id, caf::message, caf::execution_unit*) /home/awelzel/zeek/auxil/broker/caf/libcaf_core/src/actor_control_block.cpp:22
      #1 0x7f391549c383 in do_run /home/awelzel/zeek/auxil/broker/caf/libcaf_core/src/actor_clock.cpp:79
      #2 0x7f391549c383 in run /home/awelzel/zeek/auxil/broker/caf/libcaf_core/src/actor_clock.cpp:60
      #3 0x7f3915783a0d in caf::action::run() /home/awelzel/zeek/auxil/broker/caf/libcaf_core/caf/action.hpp:74
      #4 0x7f3915783a0d in caf::detail::thread_safe_actor_clock::run() /home/awelzel/zeek/auxil/broker/caf/libcaf_core/src/detail/thread_safe_actor_clock.cpp:45
      #5 0x7f39157850a2 in operator() /home/awelzel/zeek/auxil/broker/caf/libcaf_core/src/detail/thread_safe_actor_clock.cpp:55
      #6 0x7f39157850a2 in operator()<caf::intrusive_ptr<caf::ref_counted> > /home/awelzel/zeek/auxil/broker/caf/libcaf_core/caf/actor_system.hpp:551
      #7 0x7f39157850a2 in __invoke_impl<void, caf::actor_system::launch_thread<caf::detail::thread_safe_actor_clock::start_dispatch_loop(caf::actor_system&)::<lambda()> >(char const*, caf::detail::thread_safe_actor_clock::start_dispatch_loop(caf::actor_system&)::<lambda()>)::<lambda(auto:34)>, caf::intrusive_ptr<caf::ref_counted> > /usr/include/c++/11/bits/invoke.h:61
      #8 0x7f39157850a2 in __invoke<caf::actor_system::launch_thread<caf::detail::thread_safe_actor_clock::start_dispatch_loop(caf::actor_system&)::<lambda()> >(char const*, caf::detail::thread_safe_actor_clock::start_dispatch_loop(caf::actor_system&)::<lambda()>)::<lambda(auto:34)>, caf::intrusive_ptr<caf::ref_counted> > /usr/include/c++/11/bits/invoke.h:96
      #9 0x7f39157850a2 in _M_invoke<0, 1> /usr/include/c++/11/bits/std_thread.h:253
      #10 0x7f39157850a2 in operator() /usr/include/c++/11/bits/std_thread.h:260
      #11 0x7f39157850a2 in _M_run /usr/include/c++/11/bits/std_thread.h:211
      #12 0x7f39147bc2b2  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc2b2)
      #13 0x7f3914443b42 in start_thread nptl/pthread_create.c:442
      #14 0x7f39144d59ff  (/lib/x86_64-linux-gnu/libc.so.6+0x1269ff)
  
  AddressSanitizer can not provide additional info.
  SUMMARY: AddressSanitizer: SEGV /home/awelzel/zeek/auxil/broker/caf/libcaf_core/src/actor_control_block.cpp:22 in caf::actor_control_block::enqueue(caf::intrusive_ptr<caf::actor_control_block>, caf::message_id, caf::message, caf::execution_unit*)
  Thread T4 (caf.clock) created by T0 here:
      #0 0x7f39168a5685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
      #1 0x7f39147bc388 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc388)
      #2 0x7f391550effd in caf::scheduler::coordinator<caf::policy::work_sharing>::start() /home/awelzel/zeek/auxil/broker/caf/libcaf_core/caf/scheduler/coordinator.hpp:62
      #3 0x7f39154cddcd in caf::actor_system::actor_system(caf::actor_system_config&) /home/awelzel/zeek/auxil/broker/caf/libcaf_core/src/actor_system.cpp:369
      #4 0x7f3915fb359e in broker::internal::endpoint_context::endpoint_context(broker::configuration&&) /home/awelzel/zeek/auxil/broker/src/endpoint.cc:1119
      #5 0x7f391600afe6 in void __gnu_cxx::new_allocator<broker::internal::endpoint_context>::construct<broker::internal::endpoint_context, broker::configuration>(broker::internal::endpoint_context*, broker::configuration&&) /usr/include/c++/11/ext/new_allocator.h:162
      #6 0x7f391600afe6 in void std::allocator_traits<std::allocator<broker::internal::endpoint_context> >::construct<broker::internal::endpoint_context, broker::configuration>(std::allocator<broker::internal::endpoint_context>&, broker::internal::endpoint_context*, broker::configuration&&) /usr/include/c++/11/bits/alloc_traits.h:516
      #7 0x7f391600afe6 in std::_Sp_counted_ptr_inplace<broker::internal::endpoint_context, std::allocator<broker::internal::endpoint_context>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<broker::configuration>(std::allocator<broker::internal::endpoint_context>, broker::configuration&&) /usr/include/c++/11/bits/shared_ptr_base.h:519
      #8 0x7f391600afe6 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<broker::internal::endpoint_context, std::allocator<broker::internal::endpoint_context>, broker::configuration>(broker::internal::endpoint_context*&, std::_Sp_alloc_shared_tag<std::allocator<broker::internal::endpoint_context> >, broker::configuration&&) /usr/include/c++/11/bits/shared_ptr_base.h:650
      #9 0x7f391600afe6 in std::__shared_ptr<broker::internal::endpoint_context, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<broker::internal::endpoint_context>, broker::configuration>(std::_Sp_alloc_shared_tag<std::allocator<broker::internal::endpoint_context> >, broker::configuration&&) /usr/include/c++/11/bits/shared_ptr_base.h:1342
      #10 0x7f391600afe6 in std::shared_ptr<broker::internal::endpoint_context>::shared_ptr<std::allocator<broker::internal::endpoint_context>, broker::configuration>(std::_Sp_alloc_shared_tag<std::allocator<broker::internal::endpoint_context> >, broker::configuration&&) /usr/include/c++/11/bits/shared_ptr.h:409
      #11 0x7f391600afe6 in std::shared_ptr<broker::internal::endpoint_context> std::allocate_shared<broker::internal::endpoint_context, std::allocator<broker::internal::endpoint_context>, broker::configuration>(std::allocator<broker::internal::endpoint_context> const&, broker::configuration&&) /usr/include/c++/11/bits/shared_ptr.h:863
      #12 0x7f391600afe6 in std::shared_ptr<broker::internal::endpoint_context> std::make_shared<broker::internal::endpoint_context, broker::configuration>(broker::configuration&&) /usr/include/c++/11/bits/shared_ptr.h:879
      #13 0x7f391600afe6 in broker::endpoint::endpoint(broker::configuration, broker::endpoint_id) /home/awelzel/zeek/auxil/broker/src/endpoint.cc:497
      #14 0x7f39160144f1 in broker::endpoint::endpoint(broker::configuration) /home/awelzel/zeek/auxil/broker/src/endpoint.cc:489
  [supervisor:STDERR] [bare] received termination signal
  [supervisor:STDERR] [bare] received signal while waiting for thread broker/Log::WRITER_ASCII, aborting all ...
  [supervisor:STDERR] [bare] received another signal while waiting for thread broker/Log::WRITER_ASCII, aborting processing
  [supervisor:STDERR] [inherit] received termination signal
  [supervisor:STDERR] [default] received termination signal
      #15 0x5611d125b640 in zeek::Broker::BrokerState::BrokerState(broker::configuration, unsigned long) /home/awelzel/zeek/src/broker/Manager.cc:167
      #16 0x5611d1245603 in void __gnu_cxx::new_allocator<zeek::Broker::BrokerState>::construct<zeek::Broker::BrokerState, broker::configuration, unsigned long&>(zeek::Broker::BrokerState*, broker::configuration&&, unsigned long&) /usr/include/c++/11/ext/new_allocator.h:162
      #17 0x5611d1245603 in void std::allocator_traits<std::allocator<zeek::Broker::BrokerState> >::construct<zeek::Broker::BrokerState, broker::configuration, unsigned long&>(std::allocator<zeek::Broker::BrokerState>&, zeek::Broker::BrokerState*, broker::configuration&&, unsigned long&) /usr/include/c++/11/bits/alloc_traits.h:516
      #18 0x5611d1245603 in std::_Sp_counted_ptr_inplace<zeek::Broker::BrokerState, std::allocator<zeek::Broker::BrokerState>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<broker::configuration, unsigned long&>(std::allocator<zeek::Broker::BrokerState>, broker::configuration&&, unsigned long&) /usr/include/c++/11/bits/shared_ptr_base.h:519
      #19 0x5611d1245603 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<zeek::Broker::BrokerState, std::allocator<zeek::Broker::BrokerState>, broker::configuration, unsigned long&>(zeek::Broker::BrokerState*&, std::_Sp_alloc_shared_tag<std::allocator<zeek::Broker::BrokerState> >, broker::configuration&&, unsigned long&) /usr/include/c++/11/bits/shared_ptr_base.h:650
      #20 0x5611d1245603 in std::__shared_ptr<zeek::Broker::BrokerState, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<zeek::Broker::BrokerState>, broker::configuration, unsigned long&>(std::_Sp_alloc_shared_tag<std::allocator<zeek::Broker::BrokerState> >, broker::configuration&&, unsigned long&) /usr/include/c++/11/bits/shared_ptr_base.h:1342
      #21 0x5611d1245603 in std::shared_ptr<zeek::Broker::BrokerState>::shared_ptr<std::allocator<zeek::Broker::BrokerState>, broker::configuration, unsigned long&>(std::_Sp_alloc_shared_tag<std::allocator<zeek::Broker::BrokerState> >, broker::configuration&&, unsigned long&) /usr/include/c++/11/bits/shared_ptr.h:409
      #22 0x5611d1245603 in std::shared_ptr<zeek::Broker::BrokerState> std::allocate_shared<zeek::Broker::BrokerState, std::allocator<zeek::Broker::BrokerState>, broker::configuration, unsigned long&>(std::allocator<zeek::Broker::BrokerState> const&, broker::configuration&&, unsigned long&) /usr/include/c++/11/bits/shared_ptr.h:863
      #23 0x5611d1245603 in std::shared_ptr<zeek::Broker::BrokerState> std::make_shared<zeek::Broker::BrokerState, broker::configuration, unsigned long&>(broker::configuration&&, unsigned long&) /usr/include/c++/11/bits/shared_ptr.h:879
      #24 0x5611d1245603 in zeek::Broker::Manager::InitPostScript() /home/awelzel/zeek/src/broker/Manager.cc:427
      #25 0x5611d1943a51 in zeek::detail::setup(int, char**, zeek::Options*) /home/awelzel/zeek/src/zeek-setup.cc:863
      #26 0x5611d2647407 in main /home/awelzel/zeek/src/main.cc:57
      #27 0x7f39143d8d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
  
  ==108940==ABORTING
  >>>
  >>> process 108938 failed with exitcode 1: zeek -j -b /home/awelzel/zeek/testing/btest/.tmp/supervisor.config-bare-mode/config-bare-mode.zeek