PRs accepted by repos during Hacktoberfest now marked as "Ineligible Repository"

[Fum]

Describe the bug
Pull requests accepted by repositories participating in Hacktoberfest during Oct 1-31 will become marked as "Ineligible Repository" if the repository removes their Hacktoberfest label after Oct 31.

To Reproduce
Steps to reproduce the behavior:

1. During Oct 1-31 submit a pull request to a repository with the Hacktoberfest label.
2. The pull request is merged BUT the pull request itself is not marked as hacktoberfest-accepted. However, it still counts as a pull request submitted to a participating repository because the repository has the Hacktoberfest label.
3. Under the user profile the pull request appears as "In Review". It is now waiting to pass the 14 day review period.
4. After Hacktoberfest has ended (From Nov 1 onwards) the project maintainer removes the Hacktoberfest label from the repo.
5. Under the user profile, the pull request is now marked as "Ineligible Repository".
6. Re-adding the Hacktoberfest label to the repo or marking the pull request as hacktoberfest-accepted does NOT have any effect. The pull request is still marked as "Ineligible Repository".

Expected behavior

• I would expect that pull requests submitted and accepted during Oct 1 - 31 would still count towards the users valid pull requests even if the Hacktoberfest label is removed from the participating repository.
• I would also expect that re-adding the Hacktoberfest label to the repo or adding hacktoberfest-accepted to the pull request would allow it to become valid.

Screenshots
This unexpected behaviour described has affected pull requests I submitted. I had 7 pull requests submitted and merged during Hacktoberfest but 4/7 are showing now showing as "Ineligible Repository". On the 3 pull requests that are still valid, I noticed the participating repo still has the Hacktoberfest label (And I assume the label was never removed).

I don't have a screenshot of my Hacktoberfest profile when it showed I completed at least 4 pull requests but here is the email confirming I completed the challenge:

Here is the status of my profile now:

Additional context
I realised this was a problem because on one of the repositories I maintain that participated in Hacktoberfest, an author mentioned one of their pull requests is not counting. I remembered I removed the Hacktoberfest label from the repo a few hours back.
I checked my own profile and saw I was also having this problem.

If this is an unexpected behaviour across all repositories that participated in Hacktoberfest, it means many pull requests will become marked as "Ineligible Repository" if the pull requests was still under review after Hacktoberfest ended and the label is removed from the repository.

And people will not realise this unless they checked their profile. (I didn't receive any email alert for this).

Happy to provide more info if requested.

[Matt Cowley]

Unfortunately, this is not a bug in the app, this is intended behaviour. Similar to how a maintainer can remove their approval or the hacktoberfest-accepted label to make a PR no longer count, the hacktoberfest topic can also be removed to opt-out the repository, invalidating all PRs in it.

As Hacktoberfest has now ended, no PRs can enter the review period now, only existing ones can remain it in. As such, if the topic is removed and later readded, the PRs affected will remain invalid as they cannot start the review period again.

We'll keep this in mind for next year, as this is certainly an edge case we hadn't considered when we rushed to roll out the opt-in changes for next year but isn't a bug in the app that we'll be fixing for this year.

If you have any questions, [e-mail] is the place to contact :)

[Fum]

@MattIPv4 Thanks for your response.

There's a few things I want to clarify:

• I did not intend to invalidate PRs already submitted to the repository. I did not realise removing the Hacktoberfest label after Oct 31st would invalidate all PRs submitted before the 31st. According to the rules for Hacktoberfest, only marking a PR as "invalid" or "spam" means it will become valid.
• The problem now is any PRs submitted to a repo that participated in Hacktoberfest would be showing now as "Ineligible Repository" just because the repo maintainer thought it was okay to remove the Hacktoberfest label after Oct 31st. As far as I'm aware I could not see anywhere in the rules stating the Hacktoberfest label must remain even after Hacktoberfest has ended.
• If removing the Hacktoberfest label from the repo after Oct 31st is turning the PR from under review to an an Ineligible submission shouldn't there be a way to reverse this? I maintained a repository that had many PRs submitted for Hacktoberfest and because I removed the Hacktoberfest label after Oct 31st without knowing the consequences it means anyone who submitted a PR to the repo will have it showing now as "Ineligible Repository". And there is no way to reverse this?

Would appreciate a response on this. I can also raise an issue at the email address provided if required. Thanks.

[Matt Cowley]

According to the rules for Hacktoberfest, only marking a PR as "invalid" or "spam" means it will become valid.

As far as I'm aware I could not see anywhere in the rules stating the Hacktoberfest label must remain even after Hacktoberfest has ended.

Whilst a spam or invalid label will move the PR to the Invalid state, a PR must also actively match the criteria noted on the website in many places (the rules) to be considered valid and be able to be in the In Review state:

PRs count if:
Submitted during the month of October AND (
 ​ ​The PR is labelled as hacktoberfest-accepted by a maintainer OR
 ​ ​Submitted in a repo with the hacktoberfest topic AND (
 ​ ​ ​ ​The PR is merged OR
 ​ ​ ​ ​The PR has been approved
 ​ ​)
)

If a PR doesn't fit these criteria at any point, it moves into the relevant state as covered at the bottom of the Hacktoberfest profile. Nowhere on the site do we ever say that the PR needs to match these criteria only once, it must always match these criteria.

And there is no way to reverse this?

Whilst Hacktoberfest is active (Oct. 1 - 31) a PR can enter the In Review state at any time, as many times as needed. However, once Hacktoberfest has ended PRs can no longer enter the In Review period. This restriction exists so that Hacktoberfest doesn't continue on infinitely, as each time a PR enters that state the 14 day review period restarts.

---

Ultimately, this is a logic flaw in how the rules for Hacktoberfest were constructed this year, an edge case that we missed when we rushed to get these rule changes out earlier in the month. Unfortuntaly, those rules are now set in stone for this year and can't be changed, but we'll absolutely make sure we cover this for next year so it doesn't happen again. :)

[Fum]

Okay I understand its an oversight but unfortunately many people will be affected by this and will fail to complete the challenge as a result :(
Thanks either way for your responses.

[Christine]

How can something be intended behaviour and a logic flaw at the same time? I think Hacktoberfest should at least acknowledge that this is a bug, even if it's not going to be fixed until next year. Emailing [e-mail] just results in being told that this is the intended behaviour.

Another thing to consider for next year, is that the 14 day review period doesn't make sense after a PR is merged. After a PR is merged, it is considered done. Merging a PR is an active action done by the maintainer; if they were going to add an "invalid" tag, they would have done so before merging the PR, and would not have merged in the first place. I believe the 14 day review period should start when the PR is submitted, not when the PR is merged. If that had been the case, maintainers removing the hacktoberfest tag on Nov 1 would have been less of an issue.

[Matt Cowley]

It is intended behaviour given the rules and logic set out for Hacktoberfest this year, it is not a bug in the code. However, it is a logic flaw in that we hadn't explicitly considered this edge case.

Another thing to consider for next year, is that the 14 day review period doesn't make sense after a PR is merged. After a PR is merged, it is considered done.

This is not the case. The 14 day period also gives our team time to identify and react to spam trends, blocking cheaty repositories etc. It also gives the team at GitHub time to ban spammy users. All PRs go through the same review period, no matter how they were accepted.

[Matt Cowley]

I mentioned this earlier in the thread but I'll mention it again -- if you have questions or feedback [e-mail] is the place to go, so that we have a record of it tracked for the whole team for next year.

[Joe Izzard]

Also gotta say this is really bad. Hacktoberfest should be about fostering open source projects and community. Your logic flaw (That you could fix, but are choosing not to) has kinda soured that for a lot of people, myself included. I have four PR's to one repo, 2 count, 2 don't. All because you made a logic error and don't want to correct it for those that are affected, doesn't seem to be in the Hacktoberfest spirit really, does it?

[Matt Cowley]

Making changes to the rules of the event nearly five weeks into the event running is not something we'd consider fair to do, whether it benefits some users or not. The fix we had discussed internally also involved some very risky patches to users that could result in more users ending up in a broken state and losing Hacktoberfest. The whole dev team had a vote and decided that a technical fix to this would not be appropriate, as it puts too many users at risk and we were not confident in making changes to the state machine and user states this far in.

As I noted before, if you have feedback to share about Hacktoberfest, please do so via the email, this repo is for technical discussion about the code itself, not meta-discussion about how the event runs.

[Joe Izzard]

I have emailed. The reasons why you decided not to fix it are irrelevant, it's still broken. People who should have won haven't because of a bug. Doesn't matter if that's in logic or code. If there was a logic error in shipment calculation for a project, it's a bug. So is this.

[Matt Cowley]

As I noted right at the start, nothing is actually broken, there is no bug. This is the intended behaviour based on how Hacktoberfest was implemented and run this year. This was how we decided to implement opt-in this year, with the app logic continually checking that the PR matches all the criteria the whole way through its review period.

A bug, in this context, would be if we'd intended to make opt-in a one time check and it was instead checking the whole way through. We didn't intend for that though, it was implemented as a continual check and that is what the app is doing.

However, I agree that this is a flaw in the rules and logic that we did decide to implement this year, and that is something we'll make sure to keep in mind and address next year when we revise the rules again. For example, a sensible change we can make next year would be to make opt-in a one time check for a PR, instead of an ongoing check the whole way through the review period.

[Patrick José Pereira]

It looks unfair that a PR done in october and approved can be disapproved in november (removing the tag) but not approved back (adding the tag back).

Also, it appears that this in indeed a bug, since the entire repository can be removed from hacktoberfest during november if the devs remove the hactoberfest topic in the repository (since october has ended).

If this is the desired behaviour by digital ocean, is this information available anywhere in the documentation ? Related about the tags effect in november.

[Joe Izzard]

As @patrickelectric point's out, it isn't mentioned anywhere in the documentation. Only that it need's be accepted by November 1st. No where did it mention the need for tags to remain past that date, or warn of any issues if you did remove it.

[Hieu Do]

I, too, was impacted by this and how this is not communicated well really disappointed and frustrated me. I contacted the organizers on Twitter and Linkedin but there was no response. I've left really frustrated the last 2 weeks. I participated last year and decided to involved more friends to join this year, so you could see I don't join this just because of the prizes. I'm not the one who doesn't know the spirit of this open source festivity. I was just disappointed at how the issue was closed like this, and how communication was not done well. It reminds me of how open source sometimes does not welcome you because the maintainers does not agree with the community. It really goes against the spirit of this festival.

@koenvg Now you know why this happened to me.

[Matt Cowley]

👋 Hey folks, I get y'all's frustration at this, I agree it isn't the best experience. I've made sure to note it down in our retro doc so that next year the way the program treats repository opt-in is a one-time flag and not a continual check. I've also noted down for the whole team that the exact details of the rules and their designed implementation should be clearly communicated -- unfortunately this wasn't achieved this year, mostly in part due to the rushed rules change in response to the surge in spam we saw.

Unfortunately though, from a development perspective, there isn't a fix we're going to be implementing in the code found in this repository, users have made state transitions now that we as a development team don't feel comfortable trying to forcefully revert.

I also agree that this wasn't well communicated, and this is something that needs to be shared with the whole Hacktoberfest team, not just myself and the few other developers that work on the code here. Equally, the feedback around wanting to change the rules next year so that opt-in is a one-time flag and not a continual check is feedback that should be shared with the whole team -- the code here implements it as a continual check as that's how the intended rules were set out this year.

The place to share feedback with the whole team is via the Hacktoberfest email address: [e-mail] -- GitHub issues are for bug reports etc. in the code, this is not one.

As there is no bug in the code and this conversation is heading into general program feedback, which should be directed to the above email address, I'm going to lock the thread.