Option to only post thread comment when errors are detected

Question

Option to only post thread comment when errors are detected

burgholzer opened this issue a year ago · comments

Hey 👋🏼

I was wondering whether you would be willing to introduce an option to only post thread comments when some errors have been detected?
The "no errors found" message creates quite some noise as it always causes an email/notifcation to be sent. Especially after merging a PR (where all errors have been resolved), the subsequent run on main always triggers such a message.
Maybe some people fancy that, but it would be great to have an option to turn that behavior off (and, e.g., remove the previous thread comment if there are no more errors).

Brendan · Answer 1 · Wed Aug 23 2023 22:05:54 GMT+0800 (China Standard Time)

I was wondering whether you would be willing to introduce an option to only post thread comments when some errors have been detected?

There wasn't any "LGTM" comment before addressing cpp-linter/cpp-linter#24 (in cpp-linter pkg v1.6.0). It would make sense to add a input flag to disable those (and restore previous behavior).

Personally I don't use the thread-comments feature because it is limited by the GitHub REST API (see #154 as an example that we can't currently solve). I think the only way to improve the feature is to spend money by hosting our own full-fledged GitHub Checks App or switch to bot-generated PR reviews (which also might yield duplicated comments without exhaustively crawling the PR thread's history). Currently, we only look at the first 30 comments on a PR because that's the limit for a single REST API call's payload; PR review comments are returned separately from PR thread comments.

the subsequent run on main always triggers such a message

You don't mention which events you use to trigger the cpp-linter-action. The push event does not have access to PR threads (as concluded in #154).

Lukas Burgholzer · Answer 2 · Wed Aug 23 2023 22:19:22 GMT+0800 (China Standard Time)

Thanks for the quick response!

I was wondering whether you would be willing to introduce an option to only post thread comments when some errors have been detected?

There wasn't any "LGTM" comment before addressing cpp-linter/cpp-linter#24 (in cpp-linter pkg v1.6.0). It would make sense to add a input flag to disable those (and restore previous behavior).
That would be nice 👍🏼

Personally I don't use the thread-comments feature because it is limited by the GitHub REST API (see #154 as an example that we can't currently solve). I think the only way to improve the feature is to spend money by hosting our own full-fledged GitHub Checks App or switch to bot-generated PR reviews (which also might yield duplicated comments without exhaustively crawling the PR thread's history). Currently, we only look at the first 30 comments on a PR because that's the limit for a single REST API call's payload; PR review comments are returned separately from PR thread comments.

I see. Maybe we'll just disable the feature as well. The new step summary is a great replacement anyways!

the subsequent run on main always triggers such a message

You don't mention which events you use to trigger the cpp-linter-action. The push event does not have access to PR threads (as concluded in #154).

We trigger on

on:
  push:
    branches:
      - main
  pull_request:

This leads to cpp-linter adding a comment to commits after PRs have been merged. See, for example, cda-tum/mqt-core@ba5244e#commitcomment-125027883
Although I guess that "problem" would also be solved by disabling the thread-comments feature, right?

Brendan · Answer 3 · Wed Aug 23 2023 22:26:34 GMT+0800 (China Standard Time)

I suppose we could also just update an existing comment instead of deleteing any old cpp-linter comments and posting new ones... That should relieve your email inbox a bit, that may lead to the comment getting hidden in long PR threads.

This leads to cpp-linter adding a comment to commits after PRs have been merged. See, for example, cda-tum/mqt-core@ba5244e#commitcomment-125027883
Although I guess that "problem" would also be solved by disabling the thread-comments feature, right?

You could also conditionally disable thread-comments for push events:

# only post comments for PR events
thread-comments: ${{ github.event_name == 'pull_request' }}

Lukas Burgholzer · Answer 4 · Wed Aug 23 2023 22:30:24 GMT+0800 (China Standard Time)

I suppose we could also just update an existing comment instead of deleteing any old cpp-linter comments and posting new ones... That should relieve your email inbox a bit, that may lead to the comment getting hidden in long PR threads.

I'd be fine with that. It's kind of how codecov handles it (at least in the default configuration). See https://docs.codecov.com/docs/pull-request-comments#behavior
They are essentially facing the same situation.

This leads to cpp-linter adding a comment to commits after PRs have been merged. See, for example, cda-tum/mqt-core@ba5244e#commitcomment-125027883
Although I guess that "problem" would also be solved by disabling the thread-comments feature, right?

You could also conditionally disable thread-comments for push events:
# only post comments for PR events
thread-comments: ${{ github.event_name == 'pull_request' }}

Thanks for the suggestion. Sounds reasonable!

Brendan · Answer 5 · Wed Aug 23 2023 22:37:27 GMT+0800 (China Standard Time)

Just for posterity: It should be github.event_name (without the s)

Thanks, I updated it. Can't believe I checked it before posting too?! 🤣

We wouldn't be able to detect if a comment was deleted in the REST API, but this is another reasonable approach.

Brendan · Answer 6 · Wed Aug 23 2023 22:46:52 GMT+0800 (China Standard Time)

BTW, I had begun work on a PR review feature but stopped when I realized that the clang-tidy suggestions couldn't be guaranteed to fit within the PR diffs... Couple that with clang-format suggestions (which might be duplicates based on the configured tidy-checks/style), and you get a headache. IIRC, the update vs replace situation would still apply to PR reviews.

Lukas Burgholzer · Answer 7 · Wed Aug 23 2023 22:50:13 GMT+0800 (China Standard Time)

We wouldn't be able to detect if a comment was deleted in the REST API, but this is another reasonable approach.

BTW, I had begun work on a PR review feature but stopped when I realized that the clang-tidy suggestions couldn't be guaranteed to fit within the PR diffs... Couple that with clang-format suggestions (which might be duplicates based on the configured tidy-checks/style), and you get a headache. IIRC, the update vs replace situation would still apply to PR reviews.

I totally understand that headache 🙃

What I get from the discussion here is that this issue might have revealed two areas for improvement:

having an option for not posting the LGTM comment if everything succeeded. Preferably, old comments could be deleted once everything looks good.
having an option to customize whether every run creates a new comment or updates an existing one.

Brendan · Answer 8 · Wed Aug 23 2023 23:03:09 GMT+0800 (China Standard Time)

Thanks for inquiring! This is being tracked by cpp-linter/cpp-linter#34

stale · Answer 9 · Sat Sep 23 2023 20:31:18 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Brendan · Answer 10 · Sat Sep 23 2023 21:02:27 GMT+0800 (China Standard Time)

PR cpp-linter/cpp-linter#35 still needs testing, but it should be the solution.

stale · Answer 11 · Tue Oct 24 2023 10:18:34 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Brendan · Answer 12 · Fri Nov 10 2023 07:17:21 GMT+0800 (China Standard Time)

The changes have been implemented downstream (in python src pkg). We'll start the new release cycle (& testing) soon.

I've also found a way to fake a ternary operator in github action expression syntax:

      - name: Run linter as action
        uses: cpp-linter/cpp-linter-action@latest
        id: linter
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          # only update comments in a PR thread:
          thread-comments: ${{ github.event_name == 'pull_request' && 'update' || 'false' }}

Warning
This hasn't been tested yet... More info about the legitimacy of this approach found in this blog post. If this works in our testing, then the fake ternary operator will be added to the README example.

Brendan · Answer 13 · Sat Nov 11 2023 08:30:59 GMT+0800 (China Standard Time)

After some testing, the following value will work well to update existing comments on only PR threads:

thread-comments: ${{ github.event_name == 'pull_request' && 'update' }}

Using this approach will also prevent comments getting posted for commits (push events).

Brendan · Answer 14 · Sat Nov 11 2023 08:41:03 GMT+0800 (China Standard Time)

Also the new no-lgtm option defaults to true. Meaning, an outdated comment will be deleted if there are no problems to report. Users can set this to false to enable LGTM comments (as that was the intrusive default for v2.6.0 & v2.6.1).