Corrupted Double-Linked List during libtorrent CDH.torrent Distribution
neha13choudhary opened this issue · comments
libtorrent version (or branch): 1.1.5
platform/architecture: SLES 12 SP5
gcc version : gcc (SUSE Linux) 4.8.5
I was building libtorrent from scratch using Python 3.8 on SLES 12 SP5. The libtorrent build was successful but
During the CDH.torrent file distribution, I encountered a runtime error "corrupted double-linked list", and the program crashed. The error message and backtrace are as follows:
*** Error in `/usr/local/bin/python3.8': corrupted double-linked list: 0x00007f6188001780 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7731f)[0x7f619049131f]
/lib64/libc.so.6(+0x7c796)[0x7f6190496796]
/lib64/libc.so.6(+0x7cb32)[0x7f6190496b32]
/lib64/libc.so.6(+0x7d583)[0x7f6190497583]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0xace55)[0x7f618e178e55]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0xad8c9)[0x7f618e1798c9]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0xa8645)[0x7f618e174645]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0x29ed55)[0x7f618e36ad55]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0x1f11c2)[0x7f618e2bd1c2]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0x1f2236)[0x7f618e2be236]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0x1c201e)[0x7f618e28e01e]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0xc6841)[0x7f618e192841]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0x1b6986)[0x7f618e282986]
/opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9(+0xd3cff)[0x7f618e19fcff]
/lib64/libpthread.so.0(+0x971a)[0x7f6190ecc71a]
/lib64/libc.so.6(clone+0x6d)[0x7f619050a17d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:20 9440783 /usr/local/bin/python3.8
00401000-00402000 r--p 00000000 fd:20 9440783 /usr/local/bin/python3.8
00402000-00403000 rw-p 00001000 fd:20 9440783 /usr/local/bin/python3.8
00f25000-01333000 rw-p 00000000 00:00 0 [heap]
7f6178000000-7f6178021000 rw-p 00000000 00:00 0
7f6178021000-7f617c000000 ---p 00000000 00:00 0
7f617c000000-7f617c021000 rw-p 00000000 00:00 0
7f617c021000-7f6180000000 ---p 00000000 00:00 0
7f6180000000-7f6180021000 rw-p 00000000 00:00 0
7f6180021000-7f6184000000 ---p 00000000 00:00 0
7f61855e4000-7f61855f8000 r-xp 00000000 fd:20 9437575 /lib64/libresolv-2.22.so
7f61855f8000-7f61857f7000 ---p 00014000 fd:20 9437575 /lib64/libresolv-2.22.so
7f61857f7000-7f61857f8000 r--p 00013000 fd:20 9437575 /lib64/libresolv-2.22.so
7f61857f8000-7f61857f9000 rw-p 00014000 fd:20 9437575 /lib64/libresolv-2.22.so
7f61857f9000-7f61857fb000 rw-p 00000000 00:00 0
7f61857fb000-7f61857fc000 ---p 00000000 00:00 0
7f61857fc000-7f6185ffc000 rw-p 00000000 00:00 0
7f6185ffc000-7f6185ffd000 ---p 00000000 00:00 0
7f6185ffd000-7f61867fd000 rw-p 00000000 00:00 0
7f61867fd000-7f61867fe000 ---p 00000000 00:00 0
7f61867fe000-7f6186ffe000 rw-p 00000000 00:00 0
7f6186ffe000-7f6186fff000 ---p 00000000 00:00 0
7f6186fff000-7f61877ff000 rw-p 00000000 00:00 0
7f61877ff000-7f6187800000 ---p 00000000 00:00 0
7f6187800000-7f6188000000 rw-p 00000000 00:00 0
7f6188000000-7f6188021000 rw-p 00000000 00:00 0
7f6188021000-7f618c000000 ---p 00000000 00:00 0
To diagnose potential memory corruption during execution, I am rebuilding libtorrent using the flag -fsanitize=address
Reference : #207
I'm seeking assistance in identifying the root cause of this issue and finding a resolution.
Thank you.
Hey @arvidn , can you please provide any pointers why this issue might be happening? How to debug this ?
Here is cordump of the process
it seems like the crash could be related to asynchronous I/O operations, potentially within libtorrent's interaction with Boost Asio.
GNU gdb (GDB; SUSE Linux Enterprise 12) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/python3.8...
[New LWP 946872]
[New LWP 946873]
[New LWP 946759]
[New LWP 946875]
[New LWP 946877]
[New LWP 946876]
[New LWP 946874]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/bin/python3.8 /opt/cloudera/cm-agent/bin/flood'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f8c549380c7 in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f8c51ffa700 (LWP 946872))]
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.22-114.22.1.x86_64 libbz2-1-debuginfo-1.0.6-30.14.1.x86_64 libffi4-debuginfo-5.3.1+r233831-12.1.x86_64 libgcc_s1-debuginfo-12.2.1+git416-1.5.1.x86_64 libopenssl1_0_0-debuginfo-1.0.2p-3.75.1.x86_64 libstdc++6-debuginfo-12.2.1+git416-1.5.1.x86_64 libz1-debuginfo-1.2.11-11.34.1.x86_64
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f8c51ffa700 (LWP 946872) 0x00007f8c549380c7 in raise () from /lib64/libc.so.6
2 Thread 0x7f8c517f9700 (LWP 946873) 0x00007f8c553b71d0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7f8c55db5700 (LWP 946759) 0x00007f8c553b71d0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
4 Thread 0x7f8c4bfff700 (LWP 946875) 0x00007f8c553b71d0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
5 Thread 0x7f8c4affd700 (LWP 946877) 0x00007f8c553b71d0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
6 Thread 0x7f8c4b7fe700 (LWP 946876) 0x00007f8c553b71d0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7f8c50ff8700 (LWP 946874) 0x00007f8c553b71d0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f8c51ffa700 (LWP 946872))]
#0 0x00007f8c549380c7 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f8c549380c7 in raise () from /lib64/libc.so.6
#1 0x00007f8c5493949a in abort () from /lib64/libc.so.6
#2 0x00007f8c54977324 in __libc_message () from /lib64/libc.so.6
#3 0x00007f8c5497c796 in malloc_printerr () from /lib64/libc.so.6
#4 0x00007f8c5497cb32 in malloc_consolidate () from /lib64/libc.so.6
#5 0x00007f8c5497d583 in _int_free () from /lib64/libc.so.6
#6 0x00007f8c52e9de55 in boost::asio::detail::object_pool_access::destroy<boost::asio::detail::epoll_reactor::descriptor_state> (o=0x7f8c4c002ab0) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/object_pool.hpp:41
#7 boost::asio::detail::object_pool<boost::asio::detail::epoll_reactor::descriptor_state>::destroy_list (this=0x7f8c4c002a90, list=0x0) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/object_pool.hpp:131
#8 boost::asio::detail::object_pool<boost::asio::detail::epoll_reactor::descriptor_state>::~object_pool (this=0x7f8c4c002a90, __in_chrg=<optimized out>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/object_pool.hpp:73
#9 0x00007f8c52e9e8c9 in boost::asio::detail::epoll_reactor::~epoll_reactor (this=0x7f8c4c0029f0, __in_chrg=<optimized out>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/impl/epoll_reactor.ipp:63
#10 boost::asio::detail::epoll_reactor::~epoll_reactor (this=0x7f8c4c0029f0, __in_chrg=<optimized out>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/impl/epoll_reactor.ipp:69
#11 0x00007f8c52e99645 in boost::asio::detail::service_registry::destroy (service=<optimized out>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/impl/service_registry.ipp:101
#12 boost::asio::detail::service_registry::~service_registry (this=0x7f8c4c002880, __in_chrg=<optimized out>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/impl/service_registry.ipp:45
#13 boost::asio::io_service::~io_service (this=0x7f8c51ff9950, __in_chrg=<optimized out>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/impl/io_service.ipp:53
#14 libtorrent::supports_ipv6 () at broadcast_socket.cpp:172
#15 0x00007f8c5308fd55 in libtorrent::udp_socket::bind (this=this@entry=0x23e28c0, ep=..., ec=...) at udp_socket.cpp:819
#16 0x00007f8c52fe21c2 in libtorrent::aux::session_impl::open_listen_port (this=this@entry=0x23e0b00) at session_impl.cpp:2137
#17 0x00007f8c52fe3236 in libtorrent::aux::session_impl::apply_settings_pack_impl (this=0x23e0b00, pack=...) at session_impl.cpp:1687
#18 0x00007f8c52fb301e in boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::shared_ptr<libtorrent::settings_pack> >::operator() (a1=..., p=0x23e0b00, this=<synthetic pointer>)
at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/bind/mem_fn_template.hpp:165
#19 boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, boost::_bi::value<boost::shared_ptr<libtorrent::settings_pack> > >::operator()<boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::shared_ptr<libtorrent::settings_pack> >, boost::_bi::list0> (a=<synthetic pointer>..., f=<synthetic pointer>..., this=<synthetic pointer>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/bind/bind.hpp:319
#20 boost::_bi::bind_t<void, boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::shared_ptr<libtorrent::settings_pack> >, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, boost::_bi::value<boost::shared_ptr<libtorrent::settings_pack> > > >::operator() (this=<synthetic pointer>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/bind/bind_template.hpp:20
#21 boost::asio::asio_handler_invoke<boost::_bi::bind_t<void, boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::shared_ptr<libtorrent::settings_pack> >, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, boost::_bi::value<boost::shared_ptr<libtorrent::settings_pack> > > > > (function=<synthetic pointer>...) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/handler_invoke_hook.hpp:69
#22 boost_asio_handler_invoke_helpers::invoke<boost::_bi::bind_t<void, boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::shared_ptr<libtorrent::settings_pack> >, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, boost::_bi::value<boost::shared_ptr<libtorrent::settings_pack> > > >, boost::_bi::bind_t<void, boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::shared_ptr<libtorrent::settings_pack> >, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, boost::_bi::value<boost::shared_ptr<libtorrent::settings_pack> > > > > (context=<synthetic pointer>..., function=<synthetic pointer>...)
at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/handler_invoke_helpers.hpp:37
#23 boost::asio::detail::completion_handler<boost::_bi::bind_t<void, boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::shared_ptr<libtorrent::settings_pack> >, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, boost::_bi::value<boost::shared_ptr<libtorrent::settings_pack> > > > >::do_complete (owner=0x229e990, base=0x22e4410) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/completion_handler.hpp:68
#24 0x00007f8c52eb7841 in boost::asio::detail::task_io_service_operation::complete (bytes_transferred=0, ec=..., owner=..., this=<optimized out>)
at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/task_io_service_operation.hpp:38
#25 boost::asio::detail::task_io_service::do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/impl/task_io_service.ipp:372
#26 boost::asio::detail::task_io_service::run (this=0x229e990, ec=...) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/impl/task_io_service.ipp:149
#27 0x00007f8c52fa7986 in boost::asio::io_service::run (this=0x22ce2c0) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/impl/io_service.ipp:59
#28 0x00007f8c52ec4cff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x2308520) at /grid/0/jenkins/cmf/build/sles12/boost//include/boost/asio/detail/impl/posix_thread.ipp:64
#29 0x00007f8c553b271a in start_thread () from /lib64/libpthread.so.0
#30 0x00007f8c549f017d in clone () from /lib64/libc.so.6
my suspicion would be that there's something corrupting the heap somewhere, and this is the symptom. I would suggest running with address-sanitizer enabled
Hi @arvidn
SLES12 SP5 comes with C++ compiler of version C++98/03 on which -fsanitize=address flag for memory error detection using AddressSanitizer is not compatible.
Got this stacktrace when -fsanitize=address flag was added.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/bin/flood", line 5, in <module>
from flood.server import main
File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/flood/server.py", line 15, in <module>
import libtorrent
ImportError: /opt/cloudera/cm-agent/lib/libtorrent-rasterbar.so.9: undefined symbol: __asan_report_load4
Reference : #207