Schedulers stuck in cpool_delete and ets locks
Yozhig opened this issue · comments
Describe the bug
3 of 24 schedulers stuck in cpool_delete function (but at different spins). System became unresponsive (remsh doesn't work etc.), CPU usage ~300%.
(gdb) info threads
Id Target Id Frame
1 Thread 0x7f15849f5740 (LWP 81841) "beam.smp" 0x00007f1584aebe63 in select () from /lib/x86_64-linux-gnu/libc.so.6
2 Thread 0x7f154357b700 (LWP 81884) "erts_ssig_disp" 0x00007f1584f1708c in read () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f15420fb700 (LWP 81885) "erts_smsg_disp" 0x00007f1584f137b2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f153d7d7700 (LWP 81886) "erts_async_1" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f153c773700 (LWP 81888) "erts_sched_1" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f153c670700 (LWP 81889) "erts_sched_2" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f153c56d700 (LWP 81890) "erts_sched_3" 0x0000561901cf446d in backoff (n=<optimized out>) at beam/erl_alloc_util.c:3209
8 Thread 0x7f153c46a700 (LWP 81891) "erts_sched_4" 0x0000561901cf463d in backoff (n=256) at beam/erl_alloc_util.c:3209
9 Thread 0x7f153c367700 (LWP 81892) "erts_sched_5" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f153c264700 (LWP 81893) "erts_sched_6" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f153c161700 (LWP 81894) "erts_sched_7" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f1517fff700 (LWP 81895) "erts_sched_8" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f1517efc700 (LWP 81896) "erts_sched_9" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f150fdf9700 (LWP 81897) "erts_sched_10" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f1517df9700 (LWP 81898) "erts_sched_11" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
* 16 Thread 0x7f1517cf6700 (LWP 81899) "erts_sched_12" 0x0000561901cf44cd in backoff (n=<optimized out>) at beam/erl_alloc_util.c:3209
17 Thread 0x7f1517bf3700 (LWP 81900) "erts_sched_13" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
18 Thread 0x7f1517af0700 (LWP 81901) "erts_sched_14" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
19 Thread 0x7f15179ed700 (LWP 81902) "erts_sched_15" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
20 Thread 0x7f15178ea700 (LWP 81903) "erts_sched_16" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
21 Thread 0x7f15177e7700 (LWP 81904) "erts_sched_17" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
22 Thread 0x7f15176e4700 (LWP 81905) "erts_sched_18" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
23 Thread 0x7f15175e1700 (LWP 81906) "erts_sched_19" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
24 Thread 0x7f15174de700 (LWP 81907) "erts_sched_20" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
25 Thread 0x7f15173db700 (LWP 81908) "erts_sched_21" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
26 Thread 0x7f15172d8700 (LWP 81909) "erts_sched_22" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
27 Thread 0x7f15171d5700 (LWP 81910) "erts_sched_23" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
28 Thread 0x7f15170d2700 (LWP 81911) "erts_sched_24" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
29 Thread 0x7f153c05e700 (LWP 81912) "erts_dcpus_1" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
30 Thread 0x7f1516fcf700 (LWP 81913) "erts_dcpus_2" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
31 Thread 0x7f1516f7c700 (LWP 81914) "erts_dcpus_3" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
32 Thread 0x7f1516f29700 (LWP 81915) "erts_dcpus_4" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
33 Thread 0x7f1516ed6700 (LWP 81916) "erts_dcpus_5" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
34 Thread 0x7f1516e83700 (LWP 81917) "erts_dcpus_6" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
35 Thread 0x7f1516e30700 (LWP 81918) "erts_dcpus_7" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
36 Thread 0x7f1516ddd700 (LWP 81919) "erts_dcpus_8" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
37 Thread 0x7f1516d8a700 (LWP 81920) "erts_dcpus_9" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
38 Thread 0x7f1516d37700 (LWP 81921) "erts_dcpus_10" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
39 Thread 0x7f1516ce4700 (LWP 81922) "erts_dcpus_11" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
40 Thread 0x7f1516c91700 (LWP 81923) "erts_dcpus_12" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
41 Thread 0x7f1516c3e700 (LWP 81924) "erts_dcpus_13" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
42 Thread 0x7f1516beb700 (LWP 81925) "erts_dcpus_14" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
43 Thread 0x7f1516b98700 (LWP 81926) "erts_dcpus_15" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
44 Thread 0x7f1516b45700 (LWP 81927) "erts_dcpus_16" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
45 Thread 0x7f1516af2700 (LWP 81928) "erts_dcpus_17" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
46 Thread 0x7f1516a9f700 (LWP 81929) "erts_dcpus_18" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
47 Thread 0x7f1516a4c700 (LWP 81930) "erts_dcpus_19" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
48 Thread 0x7f15169f9700 (LWP 81931) "erts_dcpus_20" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
49 Thread 0x7f15169a6700 (LWP 81932) "erts_dcpus_21" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
50 Thread 0x7f1516953700 (LWP 81933) "erts_dcpus_22" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
51 Thread 0x7f1516900700 (LWP 81934) "erts_dcpus_23" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
52 Thread 0x7f15168ad700 (LWP 81935) "erts_dcpus_24" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
53 Thread 0x7f151685a700 (LWP 81936) "erts_dios_1" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
54 Thread 0x7f1516807700 (LWP 81937) "erts_dios_2" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
55 Thread 0x7f15167b4700 (LWP 81938) "erts_dios_3" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
56 Thread 0x7f1516761700 (LWP 81939) "erts_dios_4" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
57 Thread 0x7f151670e700 (LWP 81940) "erts_dios_5" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
58 Thread 0x7f15166bb700 (LWP 81941) "erts_dios_6" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
59 Thread 0x7f1516668700 (LWP 81942) "erts_dios_7" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
60 Thread 0x7f1516615700 (LWP 81943) "erts_dios_8" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
61 Thread 0x7f15165c2700 (LWP 81944) "erts_dios_9" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
62 Thread 0x7f151656f700 (LWP 81945) "erts_dios_10" 0x00007f1584aeef69 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
63 Thread 0x7f151651c700 (LWP 81946) "erts_aux_1" 0x00007f1584ae9aa6 in ppoll () from /lib/x86_64-linux-gnu/libc.so.6
64 Thread 0x7f15164c9700 (LWP 81947) "erts_poll_0" 0x00007f1584af5d96 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
Threads 7, 8 and 16 bt:
Thread 7 (Thread 0x7f153c56d700 (LWP 81890) "erts_sched_3"):
#0 0x0000561901cf446d in backoff (n=<optimized out>) at beam/erl_alloc_util.c:3209
#1 cpool_mod_mark_exp (exp=139723602313248, aptr=0x7f13e91a0030) at beam/erl_alloc_util.c:3296
#2 cpool_delete (crr=crr@entry=0x7f13efaec000, prev_allctr=0x561903086000, allctr=0x561903086000) at beam/erl_alloc_util.c:3432
#3 0x0000561901cf8d5c in cpool_fetch (size=<optimized out>, allctr=<optimized out>) at beam/erl_alloc_util.c:3588
#4 create_carrier (allctr=allctr@entry=0x561903086000, umem_sz=umem_sz@entry=296, flags=flags@entry=2) at beam/erl_alloc_util.c:4152
#5 0x0000561901cf9090 in mbc_alloc_block (blk_szp=<synthetic pointer>, size=280, allctr=0x561903086000) at beam/erl_alloc_util.c:2390
#6 mbc_alloc (allctr=0x561903086000, size=280) at beam/erl_alloc_util.c:2516
#7 0x0000561901cfe242 in erts_alcu_alloc_thr_pref (type=27690, extra=<optimized out>, size=280) at beam/erl_alloc_util.c:6100
#8 0x0000561901e13341 in erts_alloc_fnf (size=280, type=27690) at beam/erl_alloc.h:288
#9 erts_bin_nrml_alloc_fnf (size=<optimized out>) at beam/erl_binary.h:412
#10 erts_bin_nrml_alloc (size=<optimized out>) at beam/erl_binary.h:427
#11 erts_bs_append_checked (c_p=0x7f153c780d10, reg=<optimized out>, live=<optimized out>, build_size_in_bits=<optimized out>, extra_words=<optimized out>, unit=<optimized out>) at beam/erl_bits.c:1625
#12 0x00007f150d9a7a78 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7f153c46a700 (LWP 81891) "erts_sched_4"):
#0 0x0000561901cf463d in backoff (n=256) at beam/erl_alloc_util.c:3209
#1 cpool_delete (crr=crr@entry=0x7f13e91a0000, prev_allctr=0x5619030871c0, allctr=0x5619030871c0) at beam/erl_alloc_util.c:3474
#2 0x0000561901cf8d5c in cpool_fetch (size=<optimized out>, allctr=<optimized out>) at beam/erl_alloc_util.c:3588
#3 create_carrier (allctr=allctr@entry=0x5619030871c0, umem_sz=umem_sz@entry=224, flags=flags@entry=2) at beam/erl_alloc_util.c:4152
#4 0x0000561901cf9090 in mbc_alloc_block (blk_szp=<synthetic pointer>, size=208, allctr=0x5619030871c0) at beam/erl_alloc_util.c:2390
#5 mbc_alloc (allctr=0x5619030871c0, size=208) at beam/erl_alloc_util.c:2516
#6 0x0000561901cfe242 in erts_alcu_alloc_thr_pref (type=27690, extra=<optimized out>, size=208) at beam/erl_alloc_util.c:6100
#7 0x0000561901dd3ed0 in erts_alloc_fnf (size=208, type=27690) at beam/erl_alloc.h:288
#8 erts_create_magic_binary_x (destructor=0x561901dc7140 <erts_db_match_prog_destructor>, alloc_type=27690, unaligned=0, size=160) at beam/erl_binary.h:520
#9 erts_create_magic_binary (destructor=0x561901dc7140 <erts_db_match_prog_destructor>, size=160) at beam/erl_binary.h:538
#10 db_match_compile (matchexpr=matchexpr@entry=0x7f153c465a40, guards=guards@entry=0x7f153c465a48, body=body@entry=0x7f153c465a50, num_progs=num_progs@entry=1, flags=flags@entry=1, err_info=err_info@entry=0x0, freasonp=<optimized out>) at beam/erl_db_util.c:1986
#11 0x0000561901de3b3b in analyze_pattern (tb=tb@entry=0x7f1430c2f278, pattern=<optimized out>, extra_validator=extra_validator@entry=0x0, mpi=mpi@entry=0x7f153c465bb0) at beam/erl_db_tree.c:2779
#12 0x0000561901de89e2 in db_select_tree_common (p=0x5619035445e8, tb=0x7f1430c2f278, tid=139723499090514, pattern=<optimized out>, reverse=0, ret=0x7f153c465c98, stack_container=0x7f1430c2f278, iter=0x0) at beam/erl_db_tree.c:1398
#13 0x0000561901de92bc in db_select_tree (p=<optimized out>, tbl=<optimized out>, tid=<optimized out>, pattern=<optimized out>, reverse=<optimized out>, ret=<optimized out>, safety=ITER_SAFE) at beam/erl_db_tree.c:1493
#14 0x0000561901db9705 in ets_select2 (p=p@entry=0x5619035445e8, tb=0x7f1430c2f278, tid=<optimized out>, ms=<optimized out>) at beam/erl_db.c:3819
#15 0x0000561901dc36b9 in ets_select_2 (A__p=0x5619035445e8, BIF__ARGS=0x7f153c465d80, A__I=<optimized out>) at beam/erl_db.c:3800
#16 0x00007f153f8da5f4 in ?? ()
#17 0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7f1517cf6700 (LWP 81899) "erts_sched_12"):
#0 0x0000561901cf44ca in backoff (n=<optimized out>) at beam/erl_alloc_util.c:3209
#1 cpool_mod_mark (aptr=0x7f1419104030) at beam/erl_alloc_util.c:3311
#2 cpool_delete (crr=crr@entry=0x7f1419104000, prev_allctr=<optimized out>, allctr=0x56190308ffc0) at beam/erl_alloc_util.c:3439
#3 0x0000561901cf87e4 in cpool_fetch (size=<optimized out>, allctr=<optimized out>) at beam/erl_alloc_util.c:3684
#4 create_carrier (allctr=allctr@entry=0x56190308ffc0, umem_sz=umem_sz@entry=296, flags=flags@entry=2) at beam/erl_alloc_util.c:4152
#5 0x0000561901cf9090 in mbc_alloc_block (blk_szp=<synthetic pointer>, size=280, allctr=0x56190308ffc0) at beam/erl_alloc_util.c:2390
#6 mbc_alloc (allctr=0x56190308ffc0, size=280) at beam/erl_alloc_util.c:2516
#7 0x0000561901cfe242 in erts_alcu_alloc_thr_pref (type=27690, extra=<optimized out>, size=280) at beam/erl_alloc_util.c:6100
#8 0x0000561901e13341 in erts_alloc_fnf (size=280, type=27690) at beam/erl_alloc.h:288
#9 erts_bin_nrml_alloc_fnf (size=<optimized out>) at beam/erl_binary.h:412
#10 erts_bin_nrml_alloc (size=<optimized out>) at beam/erl_binary.h:427
#11 erts_bs_append_checked (c_p=0x5619034f04c8, reg=<optimized out>, live=<optimized out>, build_size_in_bits=<optimized out>, extra_words=<optimized out>, unit=<optimized out>) at beam/erl_bits.c:1625
#12 0x00007f150d9a7988 in ?? ()
#13 0x0000000000000000 in ?? ()
To Reproduce
I think it's hard to reproduce due to async nature of this state.
Expected behavior
System operates as usual, remsh and other interfaces work as expected.
Affected versions
OTP version 26.2.3
Additional context
Additional info from gdb will be available as a gist. If you need something else (except core dump), let me know. The system will be up and available for observation for some time.
I've added to the gist backtraces for all normal scheduler threads. Seems like the reason is a combination of ets locks + allocations in ets_select_2. The table in a topic:
(gdb) p {DbTableCommon} 0x7f1430c2f278
$1 = {refc = {counter = 4}, fix_count = {counter = 0}, all = {next = 0x7f1430c2fed0, prev = 0x7f1430c2e878}, owned = {next = 0x7f142c133bd0, prev = 0x7f1430c2e878}, rwlock = {rwmtx = {
mtxb = {flgs = {counter = 1610612737}, aux_scnt = 50, main_scnt = 2000, qlck = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, q = 0x7f154357c2c0}, type = ETHR_RWMUTEX_TYPE_NORMAL, rq_end = 0x7f154357c480, tdata = {
ra = 0x7, rs = 7}}}, fixlock = {mtx = {pt_mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}}, is_thread_safe = 1, type = 132, owner = 373891, heir = 1291, heir_data = 0, heir_started_interval = 0,
the_name = 253835, btid = 0x7f150fe88c30, meth = 0x561902010e00 <db_tree>, counters = {nr_of_counters = 2, is_decentralized = 0, u = {counters_ptr = {counter = 83198}, counters = {{
counter = 83198}}}}, extra_for_flxctr = "(\274\254\000\000\000\000", time = {monotonic = 0, offset = 0}, fixing_procs = 0x0, status = 132, keypos = 1, compress = 0,
continuation_ctx = 0x0}
It is a mnesia_index
Thanks for the excellent bug report!
We found a bug that caused scheduler 8 to get stuck forever (in the loop at beam/erl_alloc_util.c:3474); which in turn caused the other schedulers to get stuck waiting for it forever. We have not been able to determine how we could end up in this scenario, but with the fix in #8627 scheduler 8 would have been able to continue which in turn would have solved the whole situation.
Thanks for the fast answer! How do you think, is +Muacul 0 a reasonable workaround until the fix will be available as a part of minor release?
Yes, then the pool won't be used at all
I've set break point on ethr_atomic_read_acqb (it's inlined actually) for thread 8, step one instruction (mov 0x18(%rdi),%rax) then replaced a content of rax (which was the value of cpd2p->prev) with a value from crr->cpool.prev. After few expected sigpipes erts unfreezed. So I can say that mentioned pr fixes my case. Thank you again!
Have you used the instrument module or run runtime_tools tests (with instrument_SUITE in particular) on the node before it got stuck?
We are trying to understand how it could end up in that buggy loop.
No. Nothing special. We didn't even run our own integration tests at that time. But this service constantly do some background job (ets lookups, math and binary constructions). For vm metrics we use erlang:system_info/1, erlang:memory/0 and erlang:statistics/1. We don't overwrite alloc and other defaults in vm.args for this setup. Interesting that we recently moved this service from otp 23.3.4 to 26.2.3.
Couldn't there be a delete marker?
Delete markers are set on next and prev of a carrier at the end of cpool_delete.
If prev already had a delete marker that could only happen (I think) if we are deleting an already deleted carrier, and that should not happen.