Giters

iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.

Home Page:http://iree.dev/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

llvm-external-projects test excluded from x86-swiftshader-asan cli and are missing llvm debug symbols

KoolJBlack opened this issue · comments

commented

Encountered when submitting #8918. A bug in llvm-external-projects involed "stack use after out of scope" was not caught in the cli due to it not being tested.

Once the missing test coverage was determined, attempting to build llvm-external-projects with -DIREE_ENABLE_ASAN=ON causes the python bindings to fail to compile for those tests: test-log-failure

The python issue specifically is centered around missing asan symbols during linking:

ImportError: /tmpfs/src/github/iree/build/iree/compiler/python/iree/compiler/_mlir_libs/libIREECompilerAggregateCAPI.so: undefined symbol: __asan_option_detect_stack_use_after_return
[18:19:16 UTC] cleanup
[18:19:16 UTC] echo 'Cleaning up test environment'
Cleaning up test environment
[18:19:16 UTC] rm -rf /tmpfs/src/github/iree/build/tmp/iree_compiler_API_python_test_transforms_ireec_compiler_options_test_test_tmpdir

Note: I haven't encountered missing asan symbol issues anywhere else in IREE when enabling asan, only llvm-external-projects.

Partial solution I found was to disable python bindings (skips tests), then add additional flags to get asan to link for the rest of the llvm-external-projects.

  • new flags: -fsanitize=address shared-libasan -lpthread
  • added to: #8986

Also related issue: crashes in the bug report did not include llvm symbols: test-log-output

commented

Ideally the ASAN CI script would look just like the non-ASAN CI script, but with ASAN enabled and tests that fail under ASAN excluded. That's not the case today:

At commit 7eb87c3, we have:

The non-ASAN build calls the test.sh script directly, while the ASAN build duplicates some logic from it. When

echo "******************** llvm-external-projects tests ***********************"
cmake --build . --target check-iree-dialects -- -k 0

was added to the non-ASAN build, it was not added to the ASAN build. That's understandable, since these scripts are located in very different places.

When we finish migrating from Kokoro to Buildkite, we can watch out for differences like this. In the interim, I think we can get these CI scripts looking more alike.

Regarding

crashes in the bug report did not include llvm symbols

we should make sure that llvm-symbolizer is on the PATH for all of our builds.

commented

Fixed a few things here

  • check-iree-dialects is now covered by the ASan pipeline (still excluding the Python bindings, I did not need to edit other flags like -fsanitize=address shared-libasan -lpthread)
  • all of our pipelines now include llvm-symbolizer on the path (via our base Docker image), so crashes should be symbolized now