HEAP_PROFILE_MMAP reports allocations twice
GoogleCodeExporter opened this issue · comments
What steps will reproduce the problem?
1. Start program with HEAP_PROFILE_MMAP=true
2. Call HeapProfilerStart(...)
3. Run pprof --inuse_space on profile file
What is the expected output? What do you see instead?
Expect to see my program's allocations only.
Also see allocations from SbrkSysAllocator::Alloc and DoAllocWithArena.
Total is about twice what it should be.
What version of the product are you using? On what operating system?
1.3 and 1.4.
2.6.24-etchnhalf.1-amd64 #1 SMP Sat Aug 15 20:38:41 UTC 2009 x86_64 GNU/
Linux
export CFLAGS="-m32 -g"
export CXXFLAGS="-m32 -g"
configure --enable-static --enable-frame-pointers --host=i686
Please provide any additional information below.
It looks to me that the counter in MemoryRegionMap does not distinguish
between sbrk/mmap calls from the program under test and calls from the
HeapProfiler. When a dump is made the two are summed.
The ill effects can be mitigated by passing --ignore='DoAllocWithArena|
SbrkSysAllocator::Alloc' to pprof. The total is still too high.
Original issue reported on code.google.com by pet...@gmail.com on 29 Oct 2009 at 4:34
Hmm, can you provide a small test program that illustrates this problem? I
imagine
one that just does a simple mmap should be enough. Then we can add it to the
test
suite for perftools, and work on a fix.
I admit the HEAP_PROFILE_MMAP stuff isn't that well tested, since we don't use
it
here at Google, so it's not totally surprising there are bugs. It will be good
to
get a unittest file to test this mode better.
If you'd like to try your hand at a patch to fix this problem, that would be
great
too. :-)
Original comment by csilv...@gmail.com on 3 Nov 2009 at 3:23
- Added labels: Priority-Medium, Type-Defect
Calls to mmap from the program under test is not an issue. The problem is when
a
regular new or malloc causes TCMalloc to acquire memory pages.
#include <memory>
#include <google/heap-profiler.h>
class Alloc20MiB {
char* buffer;
public:
Alloc20MiB() {
buffer = new char[20 * 1024 * 1024];
}
~Alloc20MiB() {
delete[] buffer;
}
};
int main(int argc, char* argv[]) {
const char* name = argc > 1 ? argv[1] : "default";
HeapProfilerStart(name);
std::auto_ptr<Alloc20MiB> m0(new Alloc20MiB());
HeapProfilerDump("about to make 2nd object");
std::auto_ptr<Alloc20MiB> m1(new Alloc20MiB());
HeapProfilerDump("about to exit");
}
The .0001 profile has the correct results.
Total: 20.0 MB
20.0 100.0% 100.0% 20.0 100.0% Alloc20MiB::Alloc20MiB
0.0 0.0% 100.0% 20.0 100.0% main
0.0 0.0% 100.0% 20.0 100.0% __libc_start_main
0.0 0.0% 100.0% 20.0 100.0% _start
When the program is run with HEAP_PROFILE_MMAP=true then extra memory usage is
erroneously reported and .0001 now looks like this.
Total: 41.9 MB
20.0 47.7% 47.7% 40.1 95.5% Alloc20MiB::Alloc20MiB
20.0 47.7% 95.4% 20.0 47.7% SbrkSysAllocator::Alloc
1.9 4.6% 100.0% 1.9 4.6% DoAllocWithArena
0.0 0.0% 100.0% 41.9 100.0% main
0.0 0.0% 100.0% 20.0 47.7% ::cpp_alloc
0.0 0.0% 100.0% 0.1 0.1% AddressMap::FindCluster
0.0 0.0% 100.0% 0.1 0.1% AddressMap::Insert
0.0 0.0% 100.0% 0.1 0.1% AddressMap::New
0.0 0.0% 100.0% 0.7 1.6% HeapProfileTable::HeapProfileTable
0.0 0.0% 100.0% 0.1 0.1% HeapProfileTable::RecordAlloc
0.0 0.0% 100.0% 0.1 0.1% HeapProfileTable::RecordAllocWithStack
0.0 0.0% 100.0% 1.9 4.5% HeapProfilerStart
0.0 0.0% 100.0% 1.9 4.6% LowLevelAlloc::AllocWithArena
0.0 0.0% 100.0% 0.1 0.1% LowLevelAlloc::NewArena
0.0 0.0% 100.0% 0.1 0.1% MallocHook::InvokeNewHook
0.0 0.0% 100.0% 0.1 0.1% MemoryRegionMap::DoInsertRegionLocked
0.0 0.0% 100.0% 0.1 0.1% MemoryRegionMap::HandleSavedRegionsLocked
0.0 0.0% 100.0% 0.1 0.3% MemoryRegionMap::Init
0.0 0.0% 100.0% 0.1 0.1% MemoryRegionMap::InsertRegionLocked
0.0 0.0% 100.0% 0.1 0.1% MemoryRegionMap::MyAllocator::Allocate
0.0 0.0% 100.0% 0.1 0.1% NewHook
0.0 0.0% 100.0% 1.8 4.3% ProfilerMalloc
0.0 0.0% 100.0% 0.1 0.1% RecordAlloc
0.0 0.0% 100.0% 0.1 0.1% STL_Allocator::allocate
0.0 0.0% 100.0% 41.9 100.0% __libc_start_main
0.0 0.0% 100.0% 41.9 100.0% _start
0.0 0.0% 100.0% 0.1 0.1% std::_Rb_tree::_M_create_node
0.0 0.0% 100.0% 0.1 0.1% std::_Rb_tree::_M_get_node
0.0 0.0% 100.0% 0.1 0.1% std::_Rb_tree::_M_insert
0.0 0.0% 100.0% 0.1 0.1% std::_Rb_tree::insert_unique
0.0 0.0% 100.0% 0.1 0.1% std::set::insert
The latter report can be partially cleaned up with --ignore='DoAllocWithArena|
SbrkSysAllocator::Alloc'
Total: 41.9 MB
20.0 47.7% 47.7% 20.0 47.7% Alloc20MiB::Alloc20MiB
0.0 0.0% 47.7% 20.0 47.7% main
0.0 0.0% 47.7% 20.0 47.7% __libc_start_main
0.0 0.0% 47.7% 20.0 47.7% _start
I don't have the time to attempt a fix myself. I think that what needs to be
done
is for TCMalloc's allocators to call __sbrk rather than the sbrk version with
hooks.
In a similar manner do_mmap and co. should be called to avoid the mmap hooks.
Original comment by pet...@gmail.com on 3 Nov 2009 at 2:03
I talked it over with some folks here, who had the following to say:
"One approach could be do something similar to how HeapLeakChecker::Disabler is
implemented to tag the "internal" mmap/etc calls." I think this is similar to
your
approach with --ignore, just at a different level.
But he also said this: "An easier approach is to do nothing (just document this
better)." I have to agree it's easier. :-) How much do you need this
filtering to
actually be better? If you do the --ignore now, even though it's not perfect,
is it
good enough for your needs?
Original comment by csilv...@gmail.com on 3 Nov 2009 at 8:36
Any more word on this? If you're ok with the status quo, or if I don't hear
back, I'll close this WontFix.
Original comment by csilv...@gmail.com on 7 Jun 2010 at 10:45
It's been a few months with no answer, so I'm closing WontFix.
Original comment by csilv...@gmail.com on 2 Aug 2010 at 1:46
- Changed state: WontFix