Segfault with simple stress test of global allocator

[Kevin Vigor]

The following simple stress test fails rapidly with a segfault:

#include <vector>
#include <cstdlib>

static constexpr size_t kMinAllocSz = 800000;
static constexpr size_t kMaxAllocSz = 900000;
static constexpr unsigned kMaxLiveAlloc = 128;  // keep no more than 128 * kMaxAllocSz memory allocated.

int main(void) {
    std::vector<void *> alloc(kMaxLiveAlloc, nullptr);

    while (1) {
        const size_t ix = rand() % kMaxLiveAlloc;
        const size_t sz = (rand() % (kMaxAllocSz - kMinAllocSz)) + kMinAllocSz;

        free(alloc[ix]);
        alloc[ix] = malloc(sz);
    }
    return 0;
}

Reproduce with:

 g++ --std=c++14 -g -Wall -Werror msimple.cpp -o msimple -ldl
LD_PRELOAD=/home/kvigor/mesh/libmesh.so ./msimple

I have attempted to debug this a bit. What I have found is that in GlobalHeap::pageAlignedAlloc() I will see that the pointer returned from mh->mallocAt() == arenaBegin(), and if I check miniheapForLocked(ptr) it does not point to the original miniheap. When I later free this allocation it fails due to the same bogus miniheap pointer being found in ThreadLocalHeap::free(). Unfortunately I have not yet been able to diagnose further.

This error does not occur if I allocate fixed size objects; having some randomness in the allocation size seems to be required to trigger the issue.

(minor sidebar: what is the "locked" implied in miniheapForLocked()? free() calls it with no obvious locks held? A comment and/or renaming of the function might help.)

[Bobby Powers]

@kevin-vigor thanks for the report + reproducer! I can trigger this locally, and am looking now.

[Bobby Powers]

also - the Locked prefix on miniheapFor was vestigal - no lock is needed to look up a miniheap with that function anymore

[Bobby Powers]

Going to have to continue this tomorrow, but again thanks for such a helpful test case + debugging notes.

I've built mesh locally with ./configure --debug --no-optimize, which enables a lot of default-off debug checks. Additionally, I've checked your test case into the repo, and you can build and run it under GDB with mesh like:

./configure --debug --no-optimize
make -j5
make src/test/global-large-stress
gdb src/test/global-large-stress

There are at least 2 things going on here: we aren't properly coalescing/reusing dirty Spans in MeshableArena in this case (exclusively large object allocations), which leads to continuously allocating new spans and enlarging the arena.

Second, this error happens right around when the new allocation is 4 GiB from the start of the arena -- it sure feels like we have some int32_t math that has snuck in somewhere and we are overflowing int.

[Bobby Powers]

@kevin-vigor both issues are fixed, and I can now run the stress test for minutes on my machine. Please shout if you turn up anything else :)

[Kevin Vigor]

Thanks for quick response! I will be banging on allocator more next week, will let you know.