cmake: only use TCMALLOC when specified in build flags

[Frank Busse]

With the latest change to the build system (#1569), TCMALLOC gets pulled in when it is available on the system. I'd prefer this to be an optional dependency that gets linked only when requested.

[Cristian Cadar]

I think it should be the default, but it should be possible to turn it off. Is this not the case?

[Frank Busse]

It's just not what one would expect. (In other applications one has --enable-jemalloc, --enable-tcmalloc, ... flags.) Why would one default to a specific non-standard allocator? And what happens when we decide to support others in the future?

Is this not the case?

This should work when the flag is correctly implemented.

[Cristian Cadar]

If TCMalloc is the best-performing allocator, it should be the default when available.
I am not sure what you mean by "what happens when we decide to support others in the future?" We would still have an ordered list of preferences, the same way we have one for constraint solvers for instance.

[Frank Busse]

If TCMalloc is the best-performing allocator

Based on what benchmarks and criteria and what versions of tcmalloc/glibc? E.g. our build scripts (and Ubuntu) still use the version from gperftools and not the latest from https://github.com/google/tcmalloc. Also, TCMALLOC can be annoying at first while debugging code or running KLEE with a sanitizer, e.g.:

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7f890e0c0672 in operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_new_delete.cpp:95
    #1 0x7f890c82d032 in MallocExtension::Register(MallocExtension*) (/usr/lib/libtcmalloc.so.4+0x2d032)
    #2 0x7f890c813ab9  (/usr/lib/libtcmalloc.so.4+0x13ab9)
    #3 0x7f890e8a60fd  (/lib64/ld-linux-x86-64.so.2+0x50fd)

SUMMARY: AddressSanitizer: 8 byte(s) leaked in 1 allocation(s).

I really think it's odd to pick up a non-standard allocator. What if I use a very slow instrumented version of TCMALLOC just for debugging purposes on my system?!

[Cristian Cadar]

In my experience it works better. But I haven't run any systematic studies.
@MartinNowack @danielschemmel what is your opinion here?

[Daniel Schemmel]

If we are just talking about defaults, then KLEE should attempt for the optimal case given available resources. This is how KLEE deals with all other dependencies (e.g., we do not require --enable-z3, --enable-stp, etc).

This obviously leads to the question what we expect the optimal case to be.

• tcmalloc used to require some additional memory (it was especially slow to release allocated memory back to the OS), but was faster than the glibc allocator. I have not checked benchmarks for some time, so it might be that either one has gotten better (or worse). See, for example the benchmark done for mimalloc
• These benchmarks also show that tcmalloc is not the "best" allocator currently available.
• tcmalloc causes minor problems with asan (which can be fixed with a suppression file), but has its own debugging opportunities, e.g., https://google.github.io/tcmalloc/gwp-asan.html . It is also easy enough to disable tcmalloc by default if asan is enabled.
• If you use a very slow, instrumented version of anything as a system default, that is really on you?

Finally, I would like to propose that if tcmalloc is not expected to be better than the glibc allocator for a significant subset of use cases, we should completely remove support.

[Frank Busse]

but was faster than the glibc allocator

Many show improvements for heavily threaded code - we don't have many threads.

but has its own debugging opportunities

I'm pretty sure that's not in the old version we use (as mentioned above).