prov/util: Intercepting malloc calls can crash apps actively using those calls

[John H. Hartman]

Describe the bug
fi_domain is not thread safe for certain providers, e.g. verbs. fi_domain may call ofi_memhooks_start which patches certain memory-related syscalls. If another thread is in the middle of one of the patched syscalls it will crash upon returning.

To Reproduce
Create one or more pthreads that repeatedly call malloc while the main thread calls fi_domain with the verbs provider. From looking at the code it appears the psm3 and efa providers may also have the same problem, but I have not verified this.

Expected behavior
I expect that none of the above threads will seg fault.

Output
Here is a sanitized backtrace of the thread that crashed. You can see it was in munmap.

#0  0x00007fd54c000000 in ?? ()
#1  0x00007fda6eb95689 in new_heap () from /lib64/libc.so.6
#2  0x00007fda6eb9605d in arena_get2.part () from /lib64/libc.so.6
#3  0x00007fda6eb98c5d in tcache_init.part () from /lib64/libc.so.6
#4  0x00007fda6eb999b6 in malloc () from /lib64/libc.so.6
#5  0x0000000000639b8e in qthread_master (arg=0x7fd66d61b380)
#6  0x00007fda6f80e2de in start_thread () from /lib64/libpthread.so.0
#7  0x00007fda6ec10133 in clone () from /lib64/libc.so.6
(gdb) x/10i 0x00007fda6eb95680
   0x7fda6eb95680 <new_heap+224>:       out    %eax,$0x4c
   0x7fda6eb95682 <new_heap+226>:       sub    %ebp,%esi
   0x7fda6eb95684 <new_heap+228>:       callq  0x7fda6ec0ac10 <munmap>
   0x7fda6eb95689 <new_heap+233>:       mov    $0x3,%edx
   0x7fda6eb9568e <new_heap+238>:       mov    %rbp,%rsi
   0x7fda6eb95691 <new_heap+241>:       mov    %rbx,%rdi
   0x7fda6eb95694 <new_heap+244>:       callq  0x7fda6ec0ac40 <mprotect>
   0x7fda6eb95699 <new_heap+249>:       test   %eax,%eax
   0x7fda6eb9569b <new_heap+251>:       jne    0x7fda6eb95728 <new_heap+392>
   0x7fda6eb956a1 <new_heap+257>:       mov    %rbp,0x10(%rbx)

Here is a sanitized backtrace of the thread that simultaneously invoked fi_domain:

#0  0x00007fda6ec722af in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
#1  0x000000000054f300 in ofi_write_patch (data_size=13, address=0x7fda6ec10cc0 <mremap>, patch_data=0x737448 <intercepts+296> "I\273\200\371T")
#2  ofi_apply_patch (intercept=0x737420 <intercepts+256>)
#3  ofi_patch_function (intercept=0x737420 <intercepts+256>)
#4  ofi_intercept_symbol (intercept=0x737420 <intercepts+256>)
#5  ofi_memhooks_start (monitor=<optimized out>)
#6  ofi_memhooks_start (monitor=<optimized out>)
#7  0x000000000054e2c2 in ofi_monitors_update (monitors=monitors@entry=0x7fffc04a6bc0)
#8  0x000000000054e95c in ofi_monitors_add_cache (monitors=monitors@entry=0x7fffc04a6ce0, cache=cache@entry=0x1798970)
#9  0x000000000055076e in ofi_mr_cache_init (domain=domain@entry=0x17987f0, monitors=monitors@entry=0x7fffc04a6ce0, cache=cache@entry=0x1798970)
#10 0x0000000000564984 in vrb_domain (fabric=0x167ee70, info=<optimized out>, domain=0x1798490, context=0x0)
#11 0x0000000000576e2c in fi_domain (context=0x0, domain=0x1798490, info=<optimized out>, fabric=<optimized out>)
#12 rxm_domain_open (fabric=0x165a200, info=0x16cea90, domain=0x7537b0 <ofi_domain>, context=0x0)
#13 0x00000000004c8a55 in fi_domain (fabric=0x165a200, info=0x16cea90, domain=0x7537b0 <ofi_domain>, context=0x0)
#14 0x00000000004cdae1 in init_ofiFabricDomain ()
#15 0x00000000004cc6f7 in init_ofi ()
#16 0x00000000004cc6e0 in chpl_comm_post_task_init () 
#17 0x00000000004c2d41 in chpl_rt_init (argc=4, argv=0x7fffc04a71e8) 
#18 0x00000000004b59ee in main (argc=4, argv=0x7fffc04a71e8)

At the time of the crash it was patching mremap having just patched munmap.

Environment:
This is on Linux, the provider is verbs.

[Sean Hefty]

There is no level of locking that can be added to libfabric which will avoid this problem. Disabling the memory cache or switching to the userfaultfd memory monitor are possible solutions here. I don't think we can ever make the memory patching safe against an app using the patched calls from another thread, especially if the app uses dlopen() to load libfabric. We could look at an option to install the hooks at library initialization, rather than when first used. That option may work for more apps.

[Wei Zhang]

switching to the userfaultfd memory monitor are possible solutions here.

When using userfaultd monitor, we have experienced that it did not capture all memory events, and cause data corruption.

We could look at an option to install the hooks at library initialization

Sometimes libfabric uses an imported memhook monitor (from open mpi).

[John H. Hartman]

If it's not possible to fix the problem then noting it in the fi_domain documentation should suffice.

[github-actions]

This issue is stale because it has been open 360 days with no activity. Remove stale label or comment, otherwise it will be closed in 7 days.