Sub-process coverage is not working for ruby >= 3.1

Question

Sub-process coverage is not working for ruby >= 3.1

maxat-mansurov-jetbrains opened this issue 2 years ago · comments

maxat-mansurov-jetbrains commented 2 years ago

Setup

OS: MacOS Monterey 12.4
VM: rbenv
Ruby: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-darwin21]
Gems:
- Bundler: 2.3.11
- Simplecov: 0.21.2
- Minitest: 5.15.0

Problem description

It seems that on rubies 3.1 & 3.2 the subprocess coverage in SimpleCov has stopped working. When we start the SimpleCov as following:

SimpleCov.start do
  enable_for_subprocesses(true)
end

and test code that uses the Process.fork, we always get the error "coverage measurement is already setup (RuntimeError)". In older versions of ruby (<= 3.0) the above setup worked without a problem. Presumably this is related to #1003. Could somebody please take a look? Thank you!

How to reproduce

Download the files from this gist: https://gist.github.com/maxat-mansurov-jetbrains/1fb1a5c5746c6ce3aefa70ab5ddb7f37
Run the test with $ bundle exec ruby test_fork_coverage.rb
Observe the error in the output:

$ bundle exec ruby test_fork_coverage.rb 
Run options: --seed 7638

# Running:

/Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov.rb:354:in `start': coverage measurement is already setup (RuntimeError)
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov.rb:354:in `start_coverage_with_criteria'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov.rb:343:in `start_coverage_measurement'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov.rb:59:in `start'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov/configuration.rb:243:in `block in at_fork'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov/process.rb:8:in `block in fork_with_simplecov'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov/process.rb:7:in `fork'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/simplecov-0.21.2/lib/simplecov/process.rb:7:in `fork_with_simplecov'
        from test_fork_coverage_fail.rb:11:in `test_fork'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest/test.rb:98:in `block (3 levels) in run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest/test.rb:195:in `capture_exceptions'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest/test.rb:95:in `block (2 levels) in run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:281:in `time_it'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest/test.rb:94:in `block in run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:376:in `on_signal'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest/test.rb:221:in `with_info_handler'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest/test.rb:93:in `run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:1042:in `run_one_method'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:350:in `run_one_method'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:337:in `block (2 levels) in run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:336:in `each'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:336:in `block in run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:376:in `on_signal'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:363:in `with_info_handler'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:335:in `run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:169:in `block in __run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:169:in `map'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:169:in `__run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:146:in `run'
        from /Users/maxat.mansurov/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/minitest-5.15.0/lib/minitest.rb:73:in `block in autorun'
.

Finished in 0.014372s, 69.5797 runs/s, 0.0000 assertions/s.

1 runs, 0 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for Minitest to /Users/maxat.mansurov/RubymineProjects/simplecov_fork_test/coverage. 0 / 0 LOC (100.0%) covered.

Tobias Pfeiffer · Answer 1 · Fri Jul 08 2022 23:27:50 GMT+0800 (China Standard Time)

Ugh. Thanks for the thorough report, I'm not sure when/if I'll get to it (trying to catch up a bit right now) - always fun when you add a new features and you're like "well surely this won't break" and well... then it does break 👀

maxat-mansurov-jetbrains · Answer 2 · Mon Aug 15 2022 18:39:57 GMT+0800 (China Standard Time)

Thanks for your response! Hope you'll find the time to look into this issue more :)
Will be glad to help with testing, if needed.

sven · Answer 3 · Mon Sep 05 2022 17:45:51 GMT+0800 (China Standard Time)

Even when sub-processes are not involved, this appears to be a problem. I am running Ruby 3.2.1 and just doing this in my rails_helper.rb:

require 'simplecov'
SimpleCov.start 'rails'

and this causes the same runtime error that

Failure/Error: SimpleCov.start 'rails'

RuntimeError:
  coverage measurement is already setup

However, this somehow appears to be a Rails 7 issue more than a Ruby issue. It appears to work fine for my application (running tests) until the following Rails 7 config is uncommented for me:

# Set both the `:open_timeout` and `:read_timeout` values for `:smtp` delivery method.
Rails.application.config.action_mailer.smtp_timeout = 5

Luka Lüdicke · Answer 4 · Tue Oct 04 2022 14:08:40 GMT+0800 (China Standard Time)

However, this somehow appears to be a Rails 7 issue more than a Ruby issue.

I am observing the problem with Rails 7 as well and the proposed fix is not working