simplecov-ruby / simplecov

Code coverage for Ruby with a powerful configuration library and automatic merging of coverage across test suites

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Feature Request] Don't require all branches to be covered twice in ensure-blocks

mak-dunkelziffer opened this issue · comments

I am currently turning on branch coverage for projects that already have full line coverage. I have stumbled across the following issue multiple times now. To get to 100% branch coverage, SimpleCov requires that each branch within an ensure-block is hit twice. Once if the ensure block is entered regularly and once if it is entered via an exception. However, in my opinion, this is no longer "branch coverage", but instead "path coverage", which is even harder to achieve and sometimes unnecessarily tricky to test.

One such example is the following:

def something_about_files
  ...
  # stuff that might raise
  ...
  tmpdir = Dir.mktmpdir('my_directory')
  ...
  # stuff that might raise
  ...
ensure
  FileUtils.rm_rf(tmpdir, secure: true) if tmpdir
end

The case that that the ensure-block is reached without exception AND tmpdir is falsey is pretty hard to test and I don't think this adds much to the guarantees that you get from your test suite.

Could we get a config option to turn off this "double branch coverage / path coverage" in ensure-blocks (and wherever else it might happen)? What are your opinions on this? Do you think that this type of coverage adds much to your test suite? Am I wrong for perceiving SimpleCov's handling of ensure-blocks as "stronger than branch coverage"? Do you have examples, where the current behavior is more reasonable than in my example?

Setup:

  • SimpleCov: 0.21.2
  • test framework: RSpec
  • projects: Rails projects and gems without Rails

This is how Ruby coverage is implemented. So I think you should request that feature in Ruby issue tracker before that could be implemented in simplecov.

Maybe you could still resolve it in SimpleCov if Ruby hands you sufficient meta information already. Do you know what kind of info you get for such a line in an ensure-block? If you only get then: 1, else: 1, then: 1, else: 0 this is obviously insufficient to resolve it in SimpleCov. But if these then/else results contain more information, it could be possible to detect situations where it would be reasonable to merge them together.

This issue seems to have been resolved for current versions of Ruby already.
Issue: https://bugs.ruby-lang.org/issues/16967
I tested this on Ruby 3.1.2 and the branch coverage indeed behaves as desired.