Test and CI: Sanitizers
ax3l opened this issue · comments
For this highly concurrent project, the tests should be run with sanitizers in order to spot hard-to-debug bugs early on.
At least Address sanitizer, Memory sanitizer, UndefinedBehaviour sanitizer, and probably also very helpful ControlFlowIntegrity sanitizer should be run (all with Clang; address/memory is also in GCC).
Contrary from what one knows from valgrind in the past, compiler sanitizers know the actual source code and have an extraordinary low false positive rate in all projects I tested them with so far (e.g. for memory leaks).
For examples, see:
- https://github.com/google/boringssl/blob/master/CMakeLists.txt
- openPMD/openPMD-api#421
- openPMD/openPMD-api#452
- https://github.com/openPMD/openPMD-api/blob/dev/.travis.yml
It's really quite easy, just set CXXFLAGS="-fsanitize=..." and LDFLAGS="-fsanitize=..." when compiling with clang (CXX=clang++ CC=clang) and then run all your tests and examples.
For the case of address sanitizer, I also recommend export ASAN_OPTIONS=detect_stack_use_after_return=1:detect_leaks=1:check_initialization_order=true:strict_init_order=true:detect_stack_use_after_scope=1 before running the tests for deeper inspection.
For test/ there are now the cmake-flags MSAN, ASAN, CFI, TSAN and UBSAN.
Apart from unit-tests, maybe it would be also sensible to write a stress-test to create huge random workloads, which could run a few days and check for deadlocks and memory errors.
Yep, I think that's a great idea. I am not sure if they even need to be huge, you can just try to fuzz your workfloads and add random waits instead of actual work for asynchronicity.
There is of course also a clang sanitizer for that: https://llvm.org/docs/LibFuzzer.html but drawing random numbers (probably not only from a uniform but some kind of exponential distribution) for dependency tree and "work sleep time" will probably do the job as well.
The unit tests now contain the Random-Graph test where execution order is verified by comparing the computation result from an asynchronous run with the result of a sequential run of a randomly generated workload with non-invertible hash operations ( same principle as in tasklib-benchmark ).
Another way to verify execution order is to run the cholesky example with different tiling-factors and check the results.