Task.await deadlock (task finishes but await never returns)
edwintorok opened this issue · comments
Is it valid to nest Task.await inside a Task.async, and to nest Task.async inside Task.async? (I would assume so, otherwise how would you implement a monadic bind using async+await)
I attempted to write a version of parallel_for that works on a stream, and not on an array of known size.
It looks like this (with some debugging sprinkled in), but gets stuck in Task.await with an infinite loop on multiple cores:
open Domainslib
let id = Atomic.make 0
let m = Mutex.create ()
let parallel_foreach_reduce: fold:(('b -> 'a -> 'b) -> 'b -> 'b) -> body:('a -> 'c) -> Task.pool -> ('c -> 'c -> 'c) -> 'c -> 'c =
fun ~fold ~body pool reduce init ->
let process tasks item =
let id = Atomic.fetch_and_add id 1 in
Mutex.lock m;
Printf.eprintf "%d - [" id;
List.iter (fun (level, _, id') -> Printf.eprintf "%d,=%d" level id') tasks;
Printf.eprintf "]\n";
Printf.eprintf "%d - %d tasks\n" id (List.length tasks);
flush stderr;
Mutex.unlock m;
let rec take_mergeable (level, acc) = function
| (level', task, id) :: rest when level' = level ->
Mutex.lock m;
Printf.eprintf "%d - mergeable level %d (rest [%d])\n" id level (List.length rest);
flush stderr;
Mutex.unlock m;
take_mergeable (level+1, (task,id) :: acc) rest
| rest ->
Mutex.lock m;
Printf.eprintf "%d - not mergeable, level %d (rest [%d])\n" id level (List.length rest);
flush stderr;
Mutex.unlock m;
level, acc, rest
in
let level, await_now, tasks = take_mergeable (0, []) tasks in
Mutex.lock m;
Printf.eprintf "%d - level %d, await_now [%d], tasks [%d]\n" id level (List.length await_now) (List.length tasks);
flush stderr;
Mutex.unlock m;
let task = Task.async pool @@ fun () ->
Mutex.lock m;
Printf.eprintf "%d - async start\n" id;
flush stderr;
Mutex.unlock m;
let r = List.fold_left (fun acc (e,id') ->
Mutex.lock m;
Printf.eprintf "%d - await %d\n" id id';
flush stderr;
Mutex.unlock m;
reduce acc @@ Task.await pool e) (body item) await_now in
Mutex.lock m;
Printf.eprintf "%d - async end\n" id;
Mutex.unlock m;
r
in
(level, task, id) :: tasks
in
List.fold_left (fun acc (_, e, _) ->
reduce acc @@ Task.await pool e) init @@ fold process []
let () =
let pool = Task.setup_pool ~num_additional_domains:2 in
let body _ = 4 in
let a = Array.init 1000 Fun.id in
let fold f init = Array.fold_left f init a in
let (_:int) = parallel_foreach_reduce ~fold ~body pool (+) 0 in
Task.teardown_pool pool
Looking at a sample failure on ocaml 4.12.0+domains it looks like the async task has "finished" (or at least it acquired the mutex, printed the message that it near the end, and unlocked it), but then there is an await happening later (we know it is later due to the mutex) and that never returns and just spins endlessly.
It appears that Task.ml thinks there should be more tasks to run (because Atomic.get returned None), but when it goes looking there are none.
Now the previous task that has finished should've set the atomic value with Atomic.set (either on success or failure), and the GC didn't appear to have run inbetween, so I'm puzzled to why Atomic.get finds None (unless the implementation of Atomic.get is incorrect? looking at the disassembly I don't see any lock prefixes or fence instructions).
Here is an example failure (and it fails in both native and bytecode modes):
99950 - async start
99950 - async end
99949 - async start
99949 - await 99948
99948 - async start
99948 - async end
99949 - async end
99947 - async start
99947 - await 99945
99946 - async start
99946 - async end
99945 - async start
99945 - await 99944
99944 - async start
99944 - async end
99945 - async end
99938 - async start
99938 - async end
99939 - async start
99939 - await 99937
99939 - await 99938
99932 - async end
99940 - async start
99940 - async end
99941 - async start
99941 - await 99940
99941 - async end
99942 - async start
99942 - async end
99943 - async start
99943 - await 99939
99939 - async end
99943 - await 99941
99947 - await 99946
99947 - async end
99943 - await 99942
99943 - async end
99951 - await 99947
99951 - await 99949
99951 - await 99950
See that 86458 is nearly at the end of the async task (there is not supposed to be anything blocking happening after this),
and then 86459 tries to wait for it, but gets stuck.
18095 edwin 20 0 256.3g 15308 2624 R 306.7 0.0 1:17.61 yx.exe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x000000000043d136 in camlDomainslib__Multi_channel__recv_poll_loop_387 () at lib/multi_channel.ml:131
131 lib/multi_channel.ml: No such file or directory.
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.33-20.fc34.x86_64
(gdb) thread apply all bt
Thread 6 (Thread 0x7fbb4dffd640 (LWP 18406) "yx.exe"):
#0 0x00007fbb56abba8a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1 0x00007fbb56ab52c0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#2 0x0000000000496f0d in caml_plat_wait (cond=cond@entry=0x500730 <all_domains+656>) at platform.c:76
#3 0x000000000049c5a9 in backup_thread_func (v=0x5006f0 <all_domains+592>) at domain.c:615
#4 0x00007fbb56aaf299 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fbb569d7353 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fbb4effe640 (LWP 18405) "yx.exe"):
#0 camlStdlib__random__bits_277 () at random.ml:77
#1 0x000000000046fdb0 in camlStdlib__random__intaux_283 () at random.ml:87
#2 0x000000000043d080 in camlDomainslib__Multi_channel__recv_poll_loop_387 () at lib/multi_channel.ml:123
#3 0x000000000043dcef in camlDomainslib__Task__await_240 () at lib/task.ml:47
#4 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#5 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#6 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#7 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#8 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#9 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#10 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#11 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#12 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#13 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#14 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#15 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#16 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#17 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#18 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#19 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#20 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#21 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#22 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#23 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#24 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#25 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#26 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#27 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#28 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#29 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#30 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#31 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
--Type <RET> for more, q to quit, c to continue without paging--
#32 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#33 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#34 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#35 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#36 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#37 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#38 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#39 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#40 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#41 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#42 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#43 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#44 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#45 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#46 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#47 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#48 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#49 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#50 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#51 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#52 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#53 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#54 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#55 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#56 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#57 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#58 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#59 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#60 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#61 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#62 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#63 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#64 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#65 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#66 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#67 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#68 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#69 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#70 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#71 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#72 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#73 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
--Type <RET> for more, q to quit, c to continue without paging--
#74 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#75 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#76 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#77 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#78 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#79 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#80 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#81 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#82 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#83 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#84 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#85 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#86 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#87 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#88 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#89 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#90 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#91 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#92 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#93 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#94 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#95 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#96 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#97 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#98 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#99 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#100 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#101 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#102 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#103 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#104 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#105 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#106 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#107 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#108 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#109 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#110 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#111 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#112 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#113 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#114 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#115 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
--Type <RET> for more, q to quit, c to continue without paging--
#116 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#117 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#118 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#119 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#120 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#121 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#122 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#123 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#124 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#125 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#126 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#127 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#128 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#129 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#130 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#131 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#132 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#133 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#134 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#135 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#136 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#137 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#138 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#139 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#140 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#141 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#142 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#143 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#144 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#145 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#146 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#147 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#148 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#149 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#150 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#151 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#152 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#153 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#154 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#155 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#156 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#157 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
--Type <RET> for more, q to quit, c to continue without paging--
#158 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#159 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#160 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#161 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#162 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#163 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#164 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#165 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#166 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#167 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#168 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#169 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#170 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#171 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#172 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#173 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#174 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#175 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#176 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#177 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#178 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#179 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#180 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#181 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#182 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#183 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#184 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#185 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#186 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#187 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#188 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#189 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#190 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#191 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#192 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#193 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#194 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#195 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#196 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#197 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#198 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#199 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
--Type <RET> for more, q to quit, c to continue without paging--
#200 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#201 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#202 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#203 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#204 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#205 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#206 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#207 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#208 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#209 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#210 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#211 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#212 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#213 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#214 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#215 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#216 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#217 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#218 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#219 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#220 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#221 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#222 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#223 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#224 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#225 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#226 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#227 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#228 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#229 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#230 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#231 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#232 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#233 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#234 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#235 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#236 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#237 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#238 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#239 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#240 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#241 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
--Type <RET> for more, q to quit, c to continue without paging--
#242 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#243 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#244 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#245 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#246 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#247 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#248 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#249 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#250 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#251 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#252 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#253 0x000000000043dbd1 in camlDomainslib__Task__worker_177 () at lib/task.ml:31
#254 0x0000000000457a12 in camlStdlib__domain__body_421 () at domain.ml:141
#255 <signal handler called>
#256 0x0000000000493010 in caml_callback_exn (closure=<optimized out>, arg=<optimized out>, arg@entry=1) at callback.c:165
#257 0x00000000004934c9 in caml_callback (closure=<optimized out>, arg=arg@entry=1) at callback.c:247
#258 0x000000000049cbd5 in domain_thread_func (v=0x7ffca851ecb0) at domain.c:712
#259 0x00007fbb56aaf299 in start_thread () from /lib64/libpthread.so.0
#260 0x00007fbb569d7353 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fbb4ffff640 (LWP 18404) "yx.exe"):
#0 0x00007fbb56abba8a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1 0x00007fbb56ab52c0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#2 0x0000000000496f0d in caml_plat_wait (cond=cond@entry=0x5004e0 <all_domains+64>) at platform.c:76
#3 0x000000000049c5a9 in backup_thread_func (v=0x5004a0 <all_domains>) at domain.c:615
#4 0x00007fbb56aaf299 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fbb569d7353 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fbb556b4640 (LWP 18403) "yx.exe"):
#0 0x00007fbb56abba8a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1 0x00007fbb56ab52c0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#2 0x0000000000496f0d in caml_plat_wait (cond=cond@entry=0x500608 <all_domains+360>) at platform.c:76
#3 0x000000000049c5a9 in backup_thread_func (v=0x5005c8 <all_domains+296>) at domain.c:615
#4 0x00007fbb56aaf299 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fbb569d7353 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbb566b5640 (LWP 18402) "yx.exe"):
#0 camlDomainslib__Multi_channel__recv_poll_loop_387 () at lib/multi_channel.ml:120
#1 0x000000000043dcef in camlDomainslib__Task__await_240 () at lib/task.ml:47
#2 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#3 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#4 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
--Type <RET> for more, q to quit, c to continue without paging--
#5 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#6 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#7 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#8 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#9 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#10 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#11 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#12 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#13 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#14 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#15 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#16 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#17 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#18 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#19 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#20 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#21 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#22 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#23 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#24 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#25 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#26 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#27 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#28 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#29 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#30 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#31 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#32 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#33 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#34 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#35 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#36 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#37 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#38 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#39 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#40 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#41 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#42 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#43 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#44 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#45 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#46 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
--Type <RET> for more, q to quit, c to continue without paging--
#47 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#48 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#49 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#50 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#51 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#52 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#53 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#54 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#55 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#56 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#57 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#58 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#59 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#60 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#61 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#62 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#63 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#64 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#65 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#66 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#67 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#68 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#69 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#70 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#71 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#72 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#73 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#74 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#75 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#76 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#77 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#78 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#79 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#80 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#81 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#82 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#83 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#84 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#85 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#86 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#87 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#88 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
--Type <RET> for more, q to quit, c to continue without paging--
#89 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#90 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#91 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#92 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#93 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#94 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#95 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#96 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#97 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#98 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#99 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#100 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#101 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#102 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#103 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#104 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#105 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#106 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#107 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#108 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#109 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#110 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#111 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#112 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#113 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#114 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#115 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#116 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#117 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#118 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#119 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#120 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#121 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#122 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#123 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#124 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#125 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#126 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#127 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#128 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#129 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#130 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
--Type <RET> for more, q to quit, c to continue without paging--
#131 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#132 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#133 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#134 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#135 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#136 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#137 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#138 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#139 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#140 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#141 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#142 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#143 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#144 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#145 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#146 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#147 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#148 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#149 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#150 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#151 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#152 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#153 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#154 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#155 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#156 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#157 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#158 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#159 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#160 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#161 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#162 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#163 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#164 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#165 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#166 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#167 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#168 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#169 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#170 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#171 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#172 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
--Type <RET> for more, q to quit, c to continue without paging--
#173 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#174 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#175 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#176 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#177 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#178 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#179 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#180 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#181 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#182 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#183 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#184 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#185 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#186 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#187 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#188 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#189 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#190 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#191 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#192 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#193 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#194 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#195 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#196 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#197 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#198 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#199 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#200 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#201 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#202 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#203 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#204 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#205 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#206 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#207 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#208 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#209 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#210 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#211 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#212 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#213 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#214 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
--Type <RET> for more, q to quit, c to continue without paging--
#215 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#216 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#217 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#218 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#219 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#220 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#221 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#222 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#223 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#224 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#225 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#226 0x000000000043dbd1 in camlDomainslib__Task__worker_177 () at lib/task.ml:31
#227 0x0000000000457a12 in camlStdlib__domain__body_421 () at domain.ml:141
#228 <signal handler called>
#229 0x0000000000493010 in caml_callback_exn (closure=<optimized out>, arg=<optimized out>, arg@entry=1) at callback.c:165
#230 0x00000000004934c9 in caml_callback (closure=<optimized out>, arg=arg@entry=1) at callback.c:247
#231 0x000000000049cbd5 in domain_thread_func (v=0x7ffca851ecb0) at domain.c:712
#232 0x00007fbb56aaf299 in start_thread () from /lib64/libpthread.so.0
#233 0x00007fbb569d7353 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbb568d5f80 (LWP 18389) "yx.exe"):
#0 0x000000000043d136 in camlDomainslib__Multi_channel__recv_poll_loop_387 () at lib/multi_channel.ml:131
#1 0x000000000043dcef in camlDomainslib__Task__await_240 () at lib/task.ml:47
#2 0x0000000000433482 in camlDune__exe__Yx__fun_554 () at yx.ml:45
#3 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#4 0x0000000000433316 in camlDune__exe__Yx__fun_522 () at yx.ml:40
#5 0x000000000043d9cb in camlDomainslib__Task__do_task_167 () at lib/task.ml:17
#6 0x000000000043dd01 in camlDomainslib__Task__await_240 () at lib/task.ml:48
#7 0x00000000004334e8 in camlDune__exe__Yx__fun_586 () at yx.ml:54
#8 0x0000000000444d66 in camlStdlib__list__fold_left_279 () at list.ml:121
#9 0x0000000000433667 in camlDune__exe__Yx__entry () at yx.ml:61
#10 0x00000000004301a4 in caml_program ()
#11 <signal handler called>
#12 0x000000000049f29f in caml_startup_common (argv=0x7ffca851eec8, pooling=<optimized out>, pooling@entry=0) at startup_nat.c:133
#13 0x000000000049f2d8 in caml_startup_exn (argv=<optimized out>) at startup_nat.c:137
#14 caml_main (argv=<optimized out>) at startup_nat.c:142
#15 0x000000000042fe2c in main (argc=<optimized out>, argv=<optimized out>) at main.c:41
18389 edwin 20 0 256.2g 5728 2752 R 299.3 0.0 0:57.52 yx.exe
I'm on Fedora 34, and the libc here is somewhat buggy when it comes to multicore (pthread_cond_wait/pthread_cond_signal in particular is known to be buggy (as is anything with glibc>=2.27): https://bugzilla.redhat.com/show_bug.cgi?id=1889892, but the above stacktrace doesn't show pthread_cond_wait)
Note that smaller values for the Array work fine, but in the thousands or millions it does fail as above almost always (not always at the same task id)
Platform:
Linux storm-broadband 5.14.9-200.fc34.x86_64 #1 SMP Thu Sep 30 11:55:35 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 43 bits physical, 48 bits virtual
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 113
Model name: AMD Ryzen 9 3900X 12-Core Processor
Stepping: 0
Frequency boost: enabled
CPU MHz: 3800.000
CPU max MHz: 4672.0698
CPU min MHz: 2200.0000
BogoMIPS: 7600.47
Virtualization: AMD-V
L1d cache: 384 KiB
L1i cache: 384 KiB
L2 cache: 6 MiB
L3 cache: 64 MiB
NUMA node0 CPU(s): 0-23
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full AMD retpoline, IBPB conditional, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdts
cp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popc
nt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext p
erfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt
_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsa
veerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave
_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
dune fragment:
(executable
(name yx)
(modules yx)
(libraries domainslib unix))
I tried adding some debugging into domainslib (there is probably a better way but just used printfs here): https://github.com/ocaml-multicore/domainslib/compare/master...edwintorok:debug?expand=1
And I got this snippet:
785 - async start
785 - await 784
786 - async start
786 - async end
setting 786
set 786
793 - async start
793 - await 792
792 - async start
792 - async end
setting 792
784 - async start
set 792
784 - async end
setting 784
set 784
787 - async start
787 - await 785
789 - async start
789 - await 788
788 - async start
788 - async end
setting 788
793 - async end
set 788
791 - async start
...
got none/exit for 785
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 783
got none/exit for 785
So 785 was waiting on 784, then 784 got its result set with Atomic.set, but later on Atomic.get for 784 stil returns none, which is weird. I could understand a little bit of delay, but looping over and over again it still says None (and it has even taken a mutex and made a sycall meanwhile), so I don't think its a cache coherency issue or a missing fence as I thought initially.
I tried adding a Domain.Sync.poll too, but didn't help.
Simplified form reported on ocaml-multicore tracker, seems to be an OCaml compiler/runtime issue, not just a domainslib issue.
Fixed by #51.