macOS 13.x Ventura, daemon crashes on start

Question

macOS 13.x Ventura, daemon crashes on start

paulatbipsync opened this issue 2 years ago · comments

When starting the daemon on macOS 13.0 Beta Ventura the daemon appears to crash on startup,

The following log lines are output:

objc[15801]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called. objc[15801]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Running the service in the foreground with:
vagrant dns --start --ontop
Runs fine as a workaround, no errors logged.

Robert Schulze · Answer 1 · Mon Oct 17 2022 21:56:00 GMT+0800 (China Standard Time)

Interesting. Might be an issue in one of our dependencies or ruby.
Which version of vagrant and ruby (/opt/vagrant/embedded/bin/ruby -v) are you running.
Please try to re-install the plugin - vagrant plugin uninstall vagrant-dns, vagrant plugin install vagrant-dns and attach the log of the "install" command. This gives us a list of which dependencies and versions got installed.

Felix Schäfer · Answer 2 · Mon Oct 17 2022 22:09:52 GMT+0800 (China Standard Time)

This seems to be something happening every now and then with stuff that usually works well under linux but doesn't like macOS in that regard. If it's the same issue it isn't limited to macOS 13.0 either.

See ansible/ansible#49207 for example for details, but the gist is trying to export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in your shell before you execute the crashing commands.

Paul Preece · Answer 3 · Mon Oct 17 2022 22:32:43 GMT+0800 (China Standard Time)

Interesting. Might be an issue in one of our dependencies or ruby.
Which version of vagrant and ruby (/opt/vagrant/embedded/bin/ruby -v) are you running.
Please try to re-install the plugin - vagrant plugin uninstall vagrant-dns, vagrant plugin install vagrant-dns and attach the log of the "install" command. This gives us a list of which dependencies and versions got installed.

Yes it looks like a dependency issue,

My ruby version:
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-darwin19]

I've attached the install log output. Issues persists after reinstall.

vagrant-plugin-install-vagrant-dns.log

Paul Preece · Answer 4 · Mon Oct 17 2022 22:33:19 GMT+0800 (China Standard Time)

This seems to be something happening every now and then with stuff that usually works well under linux but doesn't like macOS in that regard. If it's the same issue it isn't limited to macOS 13.0 either.

See ansible/ansible#49207 for example for details, but the gist is trying to export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in your shell before you execute the crashing commands.

Thank you! This workaround prevented the issue.

Robert Schulze · Answer 5 · Mon Oct 17 2022 23:14:05 GMT+0800 (China Standard Time)

A bit weird though, that Python and Ruby (I even found a Nix issue, mentioning the same env var) all sharing the same issue.

Fred Ngo · Answer 6 · Fri Nov 18 2022 02:39:52 GMT+0800 (China Standard Time)

Just updated to Ventura and found this problem... so no longer a Ventura Beta problem. 😢

Matthias Pigulla · Answer 7 · Fri Dec 09 2022 01:37:47 GMT+0800 (China Standard Time)

Any chance Vagrant or this plugin or ... could take care of exporting that env var?

I don't know what OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES does, but it does not sound as if it were a smart choice to set that in your .profile, right?

Jano Paetzold · Answer 8 · Fri Dec 09 2022 05:07:49 GMT+0800 (China Standard Time)

From https://stackoverflow.com/questions/73638290/python-on-mac-is-it-safe-to-set-objc-disable-initialize-fork-safety-yes-globall:

Apple changed the way fork() behaves in High Sierra (>= 10.13).

If enabled, the OBJC_DISABLE_INITIALIZE_FORK_SAFETY variable turns off the immediate crash behaviour that their newer ObjectiveC framework otherwise enforces by default, as part of this change.

Your question "is it safe to set it as a global environment variable" depends on your definition of "safe" in this context.

It's safe in the sense your computer won't burst into flames.

It's unsafe in the sense it may mask crash information that would be otherwise presented by an app that goes awry, and may allow a fork-bomb type process to crash your computer.

So if you only have one use case where setting the flag is strictly necessary, then it's best to localise its setting to just that script/scenario.

I'm not sure if this means that OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES creates the same behaviour for fork() as it had before the change?

If yes: my computer indeed did not burst into flames using older MacOS versions (and I had no fork-bomb type processes crashing my computer either). So I consider this a workaround?

According to https://stackoverflow.com/a/46593856/20727119, this issue should have been fixed in Ruby 2.4.4 and higher, but I have Ruby 2.6 and the problem still occurs.

This highly-upvoted StackOverflow reply even recommends putting the environment variable in your .profile. However, only because it is highly upvoted doesn't mean that the author knows what they're doing. I certainly still don't ¯_(ツ)_/¯

Fred Ngo · Answer 9 · Fri Dec 09 2022 22:34:54 GMT+0800 (China Standard Time)

If the behavior of fork() changed in High Sierra, I wonder why this problem is appearing only now in Ventura?

Matthias Pigulla · Answer 10 · Sat Dec 10 2022 06:38:43 GMT+0800 (China Standard Time)

Might be a different version of Ruby that is bundled with Ventura?

Robert Schulze · Answer 11 · Sat Dec 10 2022 18:15:06 GMT+0800 (China Standard Time)

AFAIK, Vagrant ships it's own Ruby.

Robert Schulze · Answer 12 · Wed Dec 21 2022 18:45:25 GMT+0800 (China Standard Time)

Finally updated to Ventura. Tested in development with a fresh install of ruby 2.7.6 (ruby-install ruby 2.7.6 -- --enable-shared) and that one works. So I believe the ruby Vagrants ships is not good on Ventura:

/opt/vagrant/embedded/bin/ruby -v
ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin19]

versus

ruby -v
ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin22]

Matthias Pigulla · Answer 13 · Thu Dec 22 2022 02:36:32 GMT+0800 (China Standard Time)

@fnordfish wow, thank you for the effort and investigation! 😍

Do you think it would make sense to @-mention folks from hashicorp/vagrant and/or raise an issue over there?

Robert Schulze · Answer 14 · Thu Dec 22 2022 05:33:07 GMT+0800 (China Standard Time)

It's all pretty weird. It also seems to have something to do with XCode 14.

Here's a small sample script that runs just fine with a fresh install of ruby 2.7.7:

gem install daemons

# try_fork_daemons.rb
require 'daemons'
opts = {
  ARGV: ARGV,
  dir_mode: :normal,
  dir: __dir__,
  log_output: true,
  log_dir: __dir__,
  backtrace: true,
  multiple: false
}

puts "script start"
Daemons.run_proc("try_fork_daemons", opts) do
  begin
    require "not-there"
  rescue LoadError
    puts "Ignore LoadError"
  end
  puts "new pid #{Process.pid}"
  loop { sleep 1 }
end

$ ruby -v try_fork_daemons.rb start
ruby 2.7.7p221 (2022-11-24 revision 36cadad643) [x86_64-darwin22]
script start
try_fork_daemons: process with pid 73355 started.

$ ruby -v try_fork_daemons.rb status
ruby 2.7.7p221 (2022-11-24 revision 36cadad643) [x86_64-darwin22]
script start
try_fork_daemons: running [pid 73355]

Runing with Vagrants Ruby

Using the daemons gem installed in vagrants gemdir (as a dependency of vagrant-dns):

$LOAD_PATH.unshift(File.join(ENV["HOME"], ".vagrant.d/gems/2.7.6/gems/daemons-1.4.1/lib"))

# ... the rest of the original try_fork_daemons.rb

$ GEMRC=/opt/vagrant/embedded/etc/gemrc GEM_HOME=/opt/vagrant/embedded/gems/2.3.4 RUBYOPT= GEM_PATH=/opt/vagrant/embedded/gems/2.3.4 RUBYLIB= /opt/vagrant/embedded/bin/ruby try_fork_daemons.rb start
script start
try_fork_daemons: process with pid 74269 started.

No luck:

$ ps -p 74269
  PID TTY           TIME CMD

Using inline bundler:

require 'bundler/inline'
gemfile(true) do
  source 'https://rubygems.org'
  gem 'daemons'
end

# ... the rest of the original try_fork_daemons.rb

$ GEMRC=/opt/vagrant/embedded/etc/gemrc GEM_HOME=/opt/vagrant/embedded/gems/2.3.4 RUBYOPT= GEM_PATH=/opt/vagrant/embedded/gems/2.3.4 RUBYLIB= /opt/vagrant/embedded/bin/ruby try_fork_daemons.rb start
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Using bundler 2.1.4
Using daemons 1.4.1
Following files may not be writable, so sudo is needed:
  /opt/vagrant/embedded/gems/2.3.4
  /opt/vagrant/embedded/gems/2.3.4/bin
  /opt/vagrant/embedded/gems/2.3.4/bin
  /opt/vagrant/embedded/gems/2.3.4/build_info
  /opt/vagrant/embedded/gems/2.3.4/cache
  /opt/vagrant/embedded/gems/2.3.4/doc
  /opt/vagrant/embedded/gems/2.3.4/extensions
  /opt/vagrant/embedded/gems/2.3.4/gems
  /opt/vagrant/embedded/gems/2.3.4/specifications
script start
try_fork_daemons: process with pid 74530 started.

This works:

$ ps -p 74530
  PID TTY           TIME CMD
74530 ??         0:00.05 try_fork_daemons

I've no idea what's really going on and what's the best approach to actually fixing it.
I still believe, that it would be nice if Vagrant could update ruby to 2.7.7 (see https://bugs.ruby-lang.org/issues/19005 and https://bugs.ruby-lang.org/issues/18912) and maybe even ship a different ruby for Ventura/XCode 14?

There's one hot-fix we could do in vagrant-dns:

In https://github.com/BerlinVagrant/vagrant-dns/blob/master/lib/vagrant-dns/service.rb#L30-L32, when we move the require statements before the daemonized block, so that everything is loaded in the parent process already.

@mpdude Yes, I was working on an issue but wanted to get a testable script first.

Robert Schulze · Answer 15 · Fri Dec 23 2022 23:26:49 GMT+0800 (China Standard Time)

I'm a bit hesitant to release it yet, but you might want to give #73 a try.

Matthias Pigulla · Answer 16 · Sat Dec 24 2022 06:18:58 GMT+0800 (China Standard Time)

How could we best try this?

/cc @janopae

Robert Schulze · Answer 17 · Sat Dec 24 2022 19:33:13 GMT+0800 (China Standard Time)

The quick and dirty way would be to directly apply the patch:
curl -s https://patch-diff.githubusercontent.com/raw/BerlinVagrant/vagrant-dns/pull/73.patch | (cd ~/.vagrant.d/gems/2.7.6/gems/vagrant-dns-2.2.2; git apply)

Or, you could clone this repo, checkout the 72-workaround-darwin22-crash branch, and build it (requires a working ruby 2.7 installation):

$ bundle install
$ bundle exec rake build
$ vagrant plugin install ./pkg/vagrant-dns-2.2.3.pre.dev1.gem

Robert Schulze · Answer 18 · Wed Dec 28 2022 17:54:22 GMT+0800 (China Standard Time)

.. or download it from here: https://dotless-downloads.s3.amazonaws.com/vagrant-dns-2.2.3.pre.dev1.gem

Jano Paetzold · Answer 19 · Mon Jan 02 2023 21:09:09 GMT+0800 (China Standard Time)

Thanks a lot, @fnordfish, I can confirm that this fixes the problem on my machine, and it didn't cause any problems yet. 👍

Robert Schulze · Answer 20 · Wed Jan 04 2023 23:40:35 GMT+0800 (China Standard Time)

Merged the work-around and released as 2.2.3

Robert Schulze · Answer 21 · Mon Mar 20 2023 22:45:28 GMT+0800 (China Standard Time)

vagrant-dns 2.2.2 works on Vagrant 2.3.5.dev