[Bug]: Flaky infinite `AddressSanitizer:DEADLYSIGNAL` loop when using gtest in github CI

[Craig Gidney]

Describe the issue

Recently I started noticing my github CI builds hanging. They were getting into an infinite loop of printing out AddressSanitizer:DEADLYSIGNAL. Initially I assumed this was my code. I didn't know where it was so I started gradually deleting the code base while preserving the failure, until I had no code and no tests left (here is the PR where I did this). All that's left is building gtest, and running it with no tests. And I still see around 10% to 20% of builds hanging with AddressSanitizer:DEADLYSIGNAL loops:

So it appears the issue is something to do with the very initial work that gtest's main method is doing, before executing any tests.

Steps to reproduce the problem

I don't know how to reproduce this on my machine. But I can trigger it in github actions.

1. Create an empty github repository with a branch named main.

2. Create a pull request to add this .github/workflows/ci.yml file:

name: ci
on:
  pull_request:
    branches:
      - main
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        attempt: [0, 1, 2, 3, 4, 5, 6, 7, 8]
    steps:
    - uses: actions/checkout@v1
    - run: |
        cd ..
        git clone https://github.com/google/googletest.git -b v1.14.0
        mkdir googletest/build && cd googletest/build
        cmake .. -DBUILD_GMOCK=OFF
        make
        sudo make install
    - run: g++ -fsanitize=address -o stim_test /usr/local/lib/libgtest_main.a /usr/local/lib/libgtest.a
    - run: ./stim_test

3. Check whether any of the 9 instances of the test action end up hanging. If none hang, re-run the checks. This is a flaky failure so you may need to try this a few times.

What version of GoogleTest are you using?

I've seen this failure both in 1.12.1 and on 1.14.0

Since this failure appears in such an older version, and it only recently started happening, it was likely triggered by a change in github side or in g++ or in address sanitization tooling rather than on the gtest side.

What operating system and version are you using?

A github actions environment using ubuntu-latest.

What compiler and version are you using?

Based on g++ -v it's 11.4.0:

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x8[6](https://github.com/quantumlib/Stim/actions/runs/8303532921/job/22727820559#step:4:7)_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x[8](https://github.com/quantumlib/Stim/actions/runs/8303532921/job/22727820559#step:4:9)6_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.4.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-XeT[9](https://github.com/quantumlib/Stim/actions/runs/8303532921/job/22727820559#step:4:10)lY/gcc-[11](https://github.com/quantumlib/Stim/actions/runs/8303532921/job/22727820559#step:4:12)-11.4.0/debian/tmp-nvptx/usr,amdgcn
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)

What build system are you using?

A single g++ command.

g++ -fsanitize=address -o stim_test /usr/local/lib/libgtest_main.a /usr/local/lib/libgtest.a

Additional context

No response

[Craig Gidney]

There are some potential indications that this is a compiler or os bug rather than a gtest bug: quantumlib/Stim#717 (comment) .

[Derek Mauro]

Sounds like this isn't a gtest issue. Re-open if you find a repro without GitHub Actions that shows the problem.

[Craig Gidney]

This issue seems serious enough to the usability of gtest that you would probably want to push on getting it fixed, even if it's not gtest's fault.

[johannes-graeter]

I am facing the same issue, it runs in a docker container. Strangely enough on some machines it works, on some it I get the mentioned error (github ci, on prem servers, laptops). Also on prem the tests pass roughly 7 out of 10 times. In the other 3 I get the error...

[Casey Waldren]

Looks like there is a workaround available: actions/runner-images#9524