Giters

ofiwg / libfabric

Open Fabric Interfaces

Home Page:http://libfabric.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

prov/opx: memory leak

hzhou opened this issue · comments

commented

Describe the bug
We are seeing memory leaks with valgrind:

$ mpirun -n 1 valgrind --leak-check=full --show-leak-kinds=all ./cpi
==726034== Memcheck, a memory error detector
==726034== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==726034== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==726034== Command: ./cpi
==726034==
Process 0 of 1 is on tiger
pi is approximately 3.1415926544231341, Error is 0.0000000008333410
wall clock time = 0.004254
==726034==
==726034== HEAP SUMMARY:
==726034==     in use at exit: 437 bytes in 5 blocks
==726034==   total heap usage: 10,479 allocs, 10,474 frees, 12,864,283 bytes allocated
==726034==
==726034== 5 bytes in 1 blocks are still reachable in loss record 1 of 5
==726034==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==726034==    by 0x741538E: strdup (strdup.c:42)
==726034==    by 0x6CBEE37: fi_opx_alloc_default_domain_attr (fi_opx_domain.c:121)
==726034==    by 0x6CE91DC: fi_opx_ini (fi_opx_init.c:467)
==726034==    by 0x6B7B077: fi_ini (fabric.c:857)
==726034==    by 0x6B7B97B: fi_getinfo (fabric.c:1094)
==726034==    by 0x4B68248: find_provider (init_provider.c:115)
==726034==    by 0x4B68248: MPIDI_OFI_find_provider (init_provider.c:71)
==726034==    by 0x4B7E549: MPIDI_OFI_init_local (ofi_init.c:564)
==726034==    by 0x4BA6B6E: MPID_Init (ch4_init.c:508)
==726034==    by 0x4B0DD29: MPII_Init_thread (mpir_init.c:230)
==726034==    by 0x4B0E729: MPIR_Init_impl (mpir_init.c:102)
==726034==    by 0x499EE71: internal_Init (c_binding.c:45877)
==726034==    by 0x499EE71: PMPI_Init (c_binding.c:45929)
==726034==
==726034== 64 bytes in 1 blocks are still reachable in loss record 2 of 5
==726034==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==726034==    by 0x6CE4175: fi_opx_alloc_default_rx_attr (fi_opx_ep.c:1418)
==726034==    by 0x6CE922A: fi_opx_ini (fi_opx_init.c:478)
==726034==    by 0x6B7B077: fi_ini (fabric.c:857)
==726034==    by 0x6B7B97B: fi_getinfo (fabric.c:1094)
==726034==    by 0x4B68248: find_provider (init_provider.c:115)
==726034==    by 0x4B68248: MPIDI_OFI_find_provider (init_provider.c:71)
==726034==    by 0x4B7E549: MPIDI_OFI_init_local (ofi_init.c:564)
==726034==    by 0x4BA6B6E: MPID_Init (ch4_init.c:508)
==726034==    by 0x4B0DD29: MPII_Init_thread (mpir_init.c:230)
==726034==    by 0x4B0E729: MPIR_Init_impl (mpir_init.c:102)
==726034==    by 0x499EE71: internal_Init (c_binding.c:45877)
==726034==    by 0x499EE71: PMPI_Init (c_binding.c:45929)
==726034==    by 0x10933C: main (in /home/hzhou/work/pull_requests/2210_hydra_pg/cpi)
==726034==
==726034== 80 bytes in 1 blocks are still reachable in loss record 3 of 5
==726034==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==726034==    by 0x6CE4312: fi_opx_alloc_default_tx_attr (fi_opx_ep.c:1459)
==726034==    by 0x6CE9210: fi_opx_ini (fi_opx_init.c:475)
==726034==    by 0x6B7B077: fi_ini (fabric.c:857)
==726034==    by 0x6B7B97B: fi_getinfo (fabric.c:1094)
==726034==    by 0x4B68248: find_provider (init_provider.c:115)
==726034==    by 0x4B68248: MPIDI_OFI_find_provider (init_provider.c:71)
==726034==    by 0x4B7E549: MPIDI_OFI_init_local (ofi_init.c:564)
==726034==    by 0x4BA6B6E: MPID_Init (ch4_init.c:508)
==726034==    by 0x4B0DD29: MPII_Init_thread (mpir_init.c:230)
==726034==    by 0x4B0E729: MPIR_Init_impl (mpir_init.c:102)
==726034==    by 0x499EE71: internal_Init (c_binding.c:45877)
==726034==    by 0x499EE71: PMPI_Init (c_binding.c:45929)
==726034==    by 0x10933C: main (in /home/hzhou/work/pull_requests/2210_hydra_pg/cpi)
==726034==
==726034== 96 bytes in 1 blocks are still reachable in loss record 4 of 5
==726034==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==726034==    by 0x6CE4551: fi_opx_alloc_default_ep_attr (fi_opx_ep.c:1508)
==726034==    by 0x6CE91F6: fi_opx_ini (fi_opx_init.c:471)
==726034==    by 0x6B7B077: fi_ini (fabric.c:857)
==726034==    by 0x6B7B97B: fi_getinfo (fabric.c:1094)
==726034==    by 0x4B68248: find_provider (init_provider.c:115)
==726034==    by 0x4B68248: MPIDI_OFI_find_provider (init_provider.c:71)
==726034==    by 0x4B7E549: MPIDI_OFI_init_local (ofi_init.c:564)
==726034==    by 0x4BA6B6E: MPID_Init (ch4_init.c:508)
==726034==    by 0x4B0DD29: MPII_Init_thread (mpir_init.c:230)
==726034==    by 0x4B0E729: MPIR_Init_impl (mpir_init.c:102)
==726034==    by 0x499EE71: internal_Init (c_binding.c:45877)
==726034==    by 0x499EE71: PMPI_Init (c_binding.c:45929)
==726034==    by 0x10933C: main (in /home/hzhou/work/pull_requests/2210_hydra_pg/cpi)
==726034==
==726034== 192 bytes in 1 blocks are still reachable in loss record 5 of 5
==726034==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==726034==    by 0x6CBEDEA: fi_opx_alloc_default_domain_attr (fi_opx_domain.c:112)
==726034==    by 0x6CE91DC: fi_opx_ini (fi_opx_init.c:467)
==726034==    by 0x6B7B077: fi_ini (fabric.c:857)
==726034==    by 0x6B7B97B: fi_getinfo (fabric.c:1094)
==726034==    by 0x4B68248: find_provider (init_provider.c:115)
==726034==    by 0x4B68248: MPIDI_OFI_find_provider (init_provider.c:71)
==726034==    by 0x4B7E549: MPIDI_OFI_init_local (ofi_init.c:564)
==726034==    by 0x4BA6B6E: MPID_Init (ch4_init.c:508)
==726034==    by 0x4B0DD29: MPII_Init_thread (mpir_init.c:230)
==726034==    by 0x4B0E729: MPIR_Init_impl (mpir_init.c:102)
==726034==    by 0x499EE71: internal_Init (c_binding.c:45877)
==726034==    by 0x499EE71: PMPI_Init (c_binding.c:45929)
==726034==    by 0x10933C: main (in /home/hzhou/work/pull_requests/2210_hydra_pg/cpi)
==726034==
==726034== LEAK SUMMARY:
==726034==    definitely lost: 0 bytes in 0 blocks
==726034==    indirectly lost: 0 bytes in 0 blocks
==726034==      possibly lost: 0 bytes in 0 blocks
==726034==    still reachable: 437 bytes in 5 blocks
==726034==         suppressed: 0 bytes in 0 blocks
==726034==
==726034== For lists of detected and suppressed errors, rerun with: -s
==726034== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Looking at the code, it appears it is missing clean up for the various default attrs -

libfabric/prov/opx/src/fi_opx_init.c

Lines 481 to 494 in a2120d6

if (fi_opx_alloc_default_domain_attr(&fi_opx_global.default_domain_attr)) {
return NULL;
}
if (fi_opx_alloc_default_ep_attr(&fi_opx_global.default_ep_attr)) {
return NULL;
}
if (fi_opx_alloc_default_tx_attr(&fi_opx_global.default_tx_attr)) {
return NULL;
}
if (fi_opx_alloc_default_rx_attr(&fi_opx_global.default_rx_attr)) {
return NULL;
}

To Reproduce
I believe this can be reproduced running valgrind on fi_info.

Environment:
Linux

commented

Separate from opx, user also reported this leak:

==24466== 32 bytes in 1 blocks are still reachable in loss record 7 of 10
==24466==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24466==    by 0x864D7E4: _dlerror_run (dlerror.c:140)
==24466==    by 0x864D050: dlopen@@GLIBC_2.2.5 (dlopen.c:87)
==24466==    by 0x71F92F9: ofi_load_dl_prov (fabric.c:692)
==24466==    by 0x71F92F9: fi_ini (fabric.c:841)
==24466==    by 0x71FA1CA: fi_getinfo (fabric.c:1094)
==24466==    by 0x53259F2: find_provider (init_provider.c:115)
==24466==    by 0x53259F2: MPIDI_OFI_find_provider (init_provider.c:71)
==24466==    by 0x5303935: MPIDI_OFI_init_local (ofi_init.c:564)
==24466==    by 0x52AD058: MPID_Init (ch4_init.c:508)
==24466==    by 0x521709B: MPII_Init_thread (mpir_init.c:230)
==24466==    by 0x5217864: MPIR_Init_impl (mpir_init.c:102)
==24466==    by 0x4F66814: internal_Init (init.c:53)
==24466==    by 0x4F66814: PMPI_Init (init.c:105)
==24466==    by 0x1088BA: main (test_mpiio.c:5)
==24466== 
==24466== 61 bytes in 1 blocks are still reachable in loss record 8 of 10
==24466==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24466==    by 0x4017880: _dl_exception_create (dl-exception.c:77)
==24466==    by 0x7BFD250: _dl_signal_error (dl-error-skeleton.c:117)
==24466==    by 0x4009812: _dl_map_object (dl-load.c:2384)
==24466==    by 0x4014EE3: dl_open_worker (dl-open.c:235)
==24466==    by 0x7BFD2DE: _dl_catch_exception (dl-error-skeleton.c:196)
==24466==    by 0x40147C9: _dl_open (dl-open.c:605)
==24466==    by 0x864CF95: dlopen_doit (dlopen.c:66)
==24466==    by 0x7BFD2DE: _dl_catch_exception (dl-error-skeleton.c:196)
==24466==    by 0x7BFD36E: _dl_catch_error (dl-error-skeleton.c:215)
==24466==    by 0x864D734: _dlerror_run (dlerror.c:162)
==24466==    by 0x864D050: dlopen@@GLIBC_2.2.5 (dlopen.c:87)

I haven't tried to reproduce it yet. Does anyone have any clues?

commented

Thank you for reporting this, we have opened an internal Jira to investigate/fix the OPX memory leaks.

commented

Hey @hzhou ! We have fixed these memory leaks internally and intend to upstream these fixes in the near future.

commented

Hello again @hzhou ! We (Cornelis) patched this issue with the release of 1.17.1 and, using valgrind, we are no longer able to reproduce the opx specific memory leaks reported above.

Would you be willing to confirm that you are no longer seeing the opx specific leaks (using either main or 1.17.1)?

commented

@hzhou reminder

commented

Sorry about that. Let me test it.

commented

We are still seeing the leak using main. Log attached.
t.log

commented

@hzhou Thanks for getting back to me and for retesting! t.log shows we've resolved the original reported memory leaks, but sadly, it seems we (opx) have introduced some new leaks. We'll take care of these new leaks and then ping here again.

commented

We are still seeing the leak using main. Log attached. t.log

Hi @hzhou. Can share more on how you generated this report file?

  • Which MPI did you build with(Intel MPI, OpenMPI, Mpich, ...)?
  • Linux distributions used for the nodes in building the program

Thanks.

commented

@hzhou we have upstreamed all of our changes in December to address these memory leaks, we are not seeing any remaining OPX memory leaks especially with intelMPI, or MPICH. We are still working on some OpenMPI leaks. could you confirm which leaks you are seeing on your end?