ALL Images need to be rebuilt with latest version of homebrew
ibuclaw opened this issue · comments
Raising here, as it seems more appropriate than cirruslabs/cirrus-ci-docs#878.
Bintray has been shut down Homebrew/discussions#691, forcing dependencies to be built from source or fail. In our case, this adds more than 30 minutes to the CI times unless brew update-reset is ran first (which still adds ~5 minutes compared to baseline).
Upstream Homebrew migrated all packages to GitHub packages in April, but Cirrus's images have not yet been rebuilt with this bundled.
big-sur-base image got update to the latest macOS 11.6 and all the up-to-date packages. Please give it a try.
big-sur-baseimage got update to the latest macOS 11.6 and all the up-to-date packages. Please give it a try.
I think that just tells me this issue still isn't fixed if only half the images have been rebuilt (there's still catalina-base which is broken).
Agreeing with @ibuclaw here, big-sur-base seems to work now, but e.g., catalina-base is still broken.
I also see that the default user on big-sure-base changed, so something else in addition to a rebuild seems to have happened, but I do not see any new commits in this repo. Given that it doesn't seem to be possible to run these macos images with the Cirrus CLI I really wish there was a better way to see what is going on inside them (other than: inspect in CI with terminal).
Put ALL in capitals. ;-)
Catalina images still not updated and as a result all OSX pipelines broke yesterday due to Homebrew/brew#12170.
Could have been avoided completely @fkorotkov.
Is it possible for you to use Big Sur? What's your use case for supporting such an old version.
macOS CI infrastructure is a huge pain mainly thanks to Apple. There are many restriction and not much support from them. Plus the bitbucket situation really messed up the things for Brew.
Is it possible for you to use Big Sur? What's your use case for supporting such an old version.
Old? Catalina? Really?
macOS CI infrastructure is a huge pain mainly thanks to Apple. There are many restriction and not much support from them. Plus the bitbucket situation really messed up the things for Brew.
One of Big sir's libc, libunwind, or libpthread is either broken, or has some hidden ABI layout/alignment change. Because a sigbus fault occurs within pthread_getspecific when you mix userland threads with unwind exceptions.
If there's the ability to ssh into a container to debug, I'm sure someone from the community would be willing to hop on and get to the bottom of it. (Does it work to start tmate from inside a running pipeline? Would that be a red-line for general usage policies?).
If the issue is only that big sur does not work for you it seems it's reasonable to investigate the reason. Lucky we recently introduced Cirrus Terminal so you can re-run a failing task with terminal access to debug the issue. See https://medium.com/cirruslabs/introducing-cirrus-terminal-a-simple-way-to-get-ssh-like-access-to-your-tasks-1def0449065d
I completely forgot about that feature, despite using it before on the FreeBSD boxes. Maybe it should be made a bit more visible than it currently is. ;-)
FYI, tracked it down to a bug in macOS that was introduced by Monterey.
Upstream bug report with Apple: https://openradar.appspot.com/FB9733712
This issue that is preventing migration to Monterey has been worked around in the meantime though, so have started to tentatively test the newer images, but there are a number of heisenbugs with them that crop up.