Incorrectly reporting 100% coverage
jagthedrummer opened this issue · comments
We're running into an issue where coverband reports data incorrectly. Here's a screenshot that illustrates the problem.
Coverband says that the class is 100% covered, but I know that I didn't make enough calls to exercise the entire controller, and there are many lines that indicate they've never been run, but that don't count as being "missed" for some reason.
This particular controller lives outside of /app/ in a directory that is treated as an "internal gem", so to speak. That directory is pulled into the Gemfile like this:
gemspec path: 'userpages'To get any coverage data at all for anything in that directory I added this to my coverband config.
config.include_gems = true
config.root_paths = ['/app/','/userpages/']
odd it looks like it is considering those lines as comments / non active lines like whitespace. I will have to take a closer look to see what is going on. Thanks for the report, guessing it is related to the gem coverage, I will see if I can recreate.. Do you know of any opensource gem that might be close to the internal gem you use... to help me recreate @jagthedrummer
@danmayer I don't know of any open source gems that would be close to what we're doing. I'm not sure if any gem that provides a controller would do it, or if there's something else peculiar about our gem.
OK I will see if I can figure out why this would happen..... not sure I will be able to debug without a way to reproduce.
@danmayer I've created a sample repo that demonstrates this problem.
https://github.com/jagthedrummer/coverband-internal-gem-demo
The project contains a directory called interal_full_mountable_gem that provides a controller and some routes for the main app. (I misspelled 'internal' and didn't catch it until I was all done. Oops.)
The internal gem is loaded by this line in Gemfile:
gemspec path: 'interal_full_mountable_gem'The routes are hooked up with this line in config/routes.rb:
mount InteralFullMountableGem::Engine, at: '/'After installing and configuring everything I ran: rake coverband:baseline
Then started the rails server and visited: http://localhost:3000/gem_test/test1
Then rake coverband:coverage
I added the coverage/ directory that I generated from coverband to the repo so you can see what I'm seeing.
Here's a screenshot of the funky coverage I'm seeing for the controller in the internal gem:
Let me know if there's anything else I can do to help you track this down.
Trying to investigate this some more, and just making notes here.
I removed tmp/coverband_coverage.json and then ran rake coverband:baseline, that produced this coverband_coverage.json file:
{
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/controllers/application_controller.rb":{
"1":1,
"4":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/helpers/application_helper.rb":{
"1":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/helpers/test_helper.rb":{
"1":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/controllers/test_controller.rb":{
"1":1,
"2":1,
"3":0,
"6":1,
"7":0
}
}None of the files in the internal gem are included in the baseline.
I immediately ran rake coverband:coverage (without staring the server or hitting any code paths). coverband_coverage.json remained unchanged, which seems expected.
I started the server, visited http://localhost:3000/gem_test/test1, then ran rake coverband:coverage again. Now coverband_coverage.json does include files from the internal gem.
{
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/controllers/application_controller.rb":{
"1":1,
"4":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/helpers/application_helper.rb":{
"1":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/helpers/test_helper.rb":{
"1":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/app/controllers/test_controller.rb":{
"1":1,
"2":1,
"3":0,
"6":1,
"7":0
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/interal_full_mountable_gem/app/controllers/interal_full_mountable_gem/gem_test_controller.rb":{
"1":1,
"3":1,
"4":1,
"5":1,
"9":1,
"6":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/interal_full_mountable_gem/app/controllers/interal_full_mountable_gem/application_controller.rb":{
"1":1,
"2":1,
"3":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/interal_full_mountable_gem/app/helpers/interal_full_mountable_gem/application_helper.rb":{
"1":1,
"2":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/interal_full_mountable_gem/app/helpers/interal_full_mountable_gem/gem_test_helper.rb":{
"1":1,
"2":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/interal_full_mountable_gem/app/views/interal_full_mountable_gem/gem_test/test1.html.erb":{
"1":1,
"3":1,
"5":1
},
"/Users/jgreen/projects/clickfunnels/coverband-internal-gem-demo/interal_full_mountable_gem/app/views/layouts/interal_full_mountable_gem/application.html.erb":{
"1":1,
"5":1,
"6":1,
"7":1,
"11":1,
"15":1,
"17":1
}
}great investigation this is on the correct path... seems like without the baseline it thinks those lines never should be executed... then when they actually are executed they end up having a few lines turn green and therefor count as 100% executed. All lines in the file that weren't executed are considered to have no expectation to be run because of the missing baseline.
OK so I have a solution for you, but I don't have a more general solution... Basically, I think this will always remain a bit of an edge case... If it is more common than I think I could try to examine all engines in a Rails app iterate through them and then load their files... but it could add a bunch of stuff that many folks don't want to track and wouldn't consider standard application code.
This is actually explained a bit in the README related to configuring rake https://github.com/danmayer/coverband#2-configuring-rake
It tries to explain that the default baseline rake task will work for MOST rails app, sometimes it requires a custom baseline. Which I think is likely the best way to cover this kind of case.... If you know of a clever way to detect this type of engine via Rails::Engine.descendants in a way to know it is a internal gem with app code, I would be up for modifying the default baseline task.
Code in this example PR: jagthedrummer/coverband-internal-gem-demo#1
Also, if you have non-code ideas on how this could be more clear in the Readme or for this case let me know... It was non-obvious to me as well that it wouldn't just work automatically.... My best thought would be I could flag any file that shows up in live data but isn't on the baseline and give a warning that the baseline isn't properly loading all the code you execute at runtime.
@danmayer Thanks for the assist here. I'm going to try your custom Rake task and see how it goes in our real project.
I think ideally it would be awesome if the default Coverband Rake task and config were able to handle additional files in an ad hoc manner. It would be great if I could just add a line to the config to tell it about extra files that should be included in the baseline and have the Rake task take it from there.
Something like this is what I'm imagining:
Coverband.configure do |configure|
# Normal config stuff goes here...
# Add files to the baseline task
config.additional_baseline_files.push(Dir.glob("my_internal_gem/app/**/*.rb"))
endok cool @jagthedrummer take a look at this PR leet me know if you think this matches your expectations. If it does I can merge and release. #92
That looks great, @danmayer! Also, your suggestion for the custom baseline task worked perfectly in our project, so I have high confidence that this will do the trick. Thanks for everything!
cool closing this out 1.5.3 was released with the new feature. Thanks!