grodowski / undercover

I have a branch which implements undercover in our ci (buildkite) pipeline. I have added undercover as a step after all specs run. I see false negative results every now and then. The changes that undercover points out needs coverage are always things someone else on the team has merged to master recently.

The command that I run is undercover --lcov <file.lcov> --compare origin/master

Steps:

Create a branch and add all logic for undercover to work.(logic - Generate all lcov files in each ci container, aggregate them into one file which is passed as an argument to undercover command. Similar to what is described here - https://github.com/grodowski/undercover/blob/master/docs/circleci_advanced.yml )
Push the branch to build on ci - it works fine so far. Undercover does not identify any non-covered changes
Pull master into my branch and push branch to origin
The ci build that was triggered shows bunch of methods as not covered. These are changes already on master, merged by team member recently.
The changes which undercover reports as uncovered are not part of what I am working on.

Hi! Thanks for using undercover and reporting the issue.

I couldn't reproduce your exact scenario based on the description, so I'd welcome more details. A sample git tree is one option.

However, I noticed a strange behaviour of undercover -c master when master has diverged from a feature branch with new commits.

#  git log --graph test-untested master
* commit a00a293bb7c2284b0436de0c331f47f6f8761352 (HEAD -> master)
|
|     change #3
|
| *   commit bbe55cd3e0435861d0411c35c5d0c2307d7f3e8b (test-untested)
| |\  Merge: 0d8ee01 0279ca5
| |/  
|/|     Merge branch 'master' into test-untested
| |
* | commit 0279ca5d7a0ae2c1e6aa13d79708dd2dff4b9015
| |
| |     untested :o #2
| |
| * commit 0d8ee017d8c37ac6dba4efab6cae5151f61d831d
|/  
|       foo #1
|
*   commit a71309fa6e5f9465ec040f36b3f4f53e64b51824 (origin/master)

Change 3 is not included in the undercover changeset. Specifically:

undercover would currently compare bbe55cd3 with a71309fa6
we'd expect it to compare bbe55cd3 with a00a293b

Not sure if this relates to your problem, but I will take a deeper look and possibly change

undercover/lib/undercover/changeset.rb

Line 76 in a71309f

repo.lookup(repo.merge_base(compare_base.to_s, head))

to something like

repo.branches.find { |b| b.name == compare_base }.target

This could be more predictable than merge_base in more complex workflows

FYI https://git-scm.com/docs/git-merge-base

@hiralmodi22 this could help with diff issues, please lmk if it does

@grodowski To simplify the branch illustration a bit;

(branch)       M - N - O - P
              /       /
(master) A - B - C - D - E - F

Briefly, we branched at B, and then ran git merge master at D.

So if we want to see what untested code blocks would be introduced by our branch, we would want to diff P vs D, correct? #68 causes us to diff P vs. F, which will include changes from E and F. The result is that Undercover would include irrelevant changes in its results.

Unless I'm missing something?

To get the correct commit to diff, I've been running git merge-base master current-branch locally, which looks like the same strategy as before #68. Granted, my workflows aren't quite as complex as in your example, so that may be a poor general-purpose solution. One suggestion, as a compromise, would be to have usage of merge_base vs rev_parse controlled via a command-line switch.

All that being said, I'm not totally sure what caused the original issue @hiralmodi22 was encountering. Could be a nuance of shallow clone? Git is more clever about that than it used to be, but could be a gap here.

Thanks for your comment @redapted. I am not sure what caused the issue either and it would be cool to get more input from @hiralmodi22.

Looking into the rev-parse versus merge-base question, I think it depends on the workflow we want undercover to best support.

In your example, if all code changes went through review (including undercover) that used merge-base, we would be sure that all additions from E and F are covered and we could safely check P w.r.t. D. On the other hand rev-parse could yield code locations that have been deleted or changed in F, but are still present in P, meaning they require tests if we merge P into F. Should we still call these false positives? I might as well be missing some obvious edge case here

There might be some confusion too, because undercover triggers on class, method and block diffs instead of individual lines.

Finally, #68 is still unreleased and I'm debating whether to just roll it back from master. Not sure if there's really that much benefit in using rev-parse.

I'm reverting #68 from master a closing this issue as not much has happened here.

Undercover picking up changes from master and reporting false negative