Not valid Github status badge for broken workflow

Question

Not valid Github status badge for broken workflow

dmvict opened this issue 2 years ago · comments

Dmytry Victorovich commented 2 years ago

Are you experiencing an issue with...

shields.io

🐞 Description

I noticed that shields.io provide status badge passing for broken workflow

The real results is :

The link to the workflow run on the screenshot. Repository where shields.io status badges are used.

The badge works well with valid workflow files. And this workflow is fixed now. Please, try to reproduce and fix the bug.

🔗 Link to the badge

https://img.shields.io/github/workflow/status/Wandalen/wTools/mod_interface?label=

💡 Possible Solution

No response

chris48s · Answer 1 · Sat Jul 02 2022 05:00:18 GMT+0800 (China Standard Time)

Our badges for github workflows are basically just "scrapers" for github's own badges so when you call https://img.shields.io/github/workflow/status/Wandalen/wTools/mod_interface/alpha it calls https://github.com/Wandalen/wTools/workflows/mod_interface/badge.svg?branch=alpha and then reports the value from that. It is essentially just a "wrapper" that allows you to use the full range of shields styles. Looks like your builds are now fixed so I now can't reproduce it but if you can break one again, check the actual github badges and see what value you've got.

Dmytry Victorovich · Answer 2 · Sat Jul 02 2022 05:24:16 GMT+0800 (China Standard Time)

I missed this screenshot in the issue description

The screenshot was maiden just before the issue was opened. So, the bug exists, and the working case for the bug is a broken workflow.

If I will have time to test this scenario, then I will report the results.
I will close the issue after approval of the correct badge work for the described case.

Caleb Cartwright · Answer 3 · Sat Jul 02 2022 06:18:11 GMT+0800 (China Standard Time)

Unfortunately I suspect this is reflective of some upstream changes GitHub have rolled out, relative to #7102

Here's what GitHub shows for some of your branches (screen grab at the bottom in case statuses change)
https://github.com/Wandalen/wTools/actions/workflows/ModuleModInterfacePush.yml/badge.svg?branch=master

https://github.com/Wandalen/wTools/workflows/mod_interface/badge.svg?branch=master

It looks like GitHub updated the docs back in https://github.com/github/docs/pull/3802/files#diff-95d58394efa588c4f92f85274259efb8ca1e10a02bd346549e65b57604c156c4. However, I think that there's either a new(ish) issue in the original, by-name endpoint, or that GitHub have just recently rolled out the changes.

As Chris noted we rely on the upstream providers to deliver information, which is what we then use. In this case the endpoint that we've always used for Workflow Status is claiming your master branch is passing, so that's what is being displayed.

I think this is something that needs to be tracked down with GitHub to get a definitive answer. We can't simply "just switch" to the workflow file name model either as that would break our existing users which is not something we want to do if GitHub plan to fix the by-name endpoint.

Arthur Neuman · Answer 4 · Sun Aug 14 2022 09:39:57 GMT+0800 (China Standard Time)

@calebcartwright What if we have it fetch legacy first and if it's a 404, fetch the new URL? (And an alternative to not use legacy at all for those with existing legacies that are now static)

Say the word and I'll put it in my existing PR

Caleb Cartwright · Answer 5 · Sun Aug 14 2022 23:50:08 GMT+0800 (China Standard Time)

That's an interesting idea @exoRift. I think I could get behind that as a tactical solution (since the badges currently appear to be broken for half the audience), so long as we keep tracking towards something more long term.

@chris48s would you have any concerns with this? I feel like it should be simple enough at the code level, and not like it would have any tangible impact on our token pool nor the traffic volume we throw github's way

chris48s · Answer 6 · Mon Aug 15 2022 04:11:52 GMT+0800 (China Standard Time)

So if we think

https://github.com/exoRift/react-fluent-mobile/workflows/Quality%20Assurance/badge.svg?branch=master is returning inaccurate or out-of-date info

and

https://github.com/exoRift/react-fluent-mobile/actions/workflows/quality_assurance.yml/badge.svg?branch=master is correct,

I think trying to serve information we believe is wrong or outdated first is the wrong call. I think there are really 2 sensible things we can do:

Option 1:

Create a new badge route that accepts the workflow file name as the param and call /actions/workflows/<WORKFLOW_FILE> to get the data
Retire/deprecate our existing route, stop serving data and show some kind of error message advising users to switch

Option 2:

Overload the existing route to take a generic :param which could be a workflow filename or name, but do it the other way round:

Try to call /actions/workflows/<param> first
Fall back to /workflows/<param> second

Option 2 has the advantage that all the existing badges out there in the wild will still "work", but if we think they are serving stale/inaccurate data then this is not necessarily much of an "advantage". I think serving an error saying "you need update your badge URL" is ultimately more useful then "here's a value that might not be right"

I guess the thing I'd like to be slightly more sure of is: what is the status of the /workflows/<WORKFLOW_NAME> endpoint we are currently using? It is certainly still returning some data. Beyond anec-data: The commit where this changed in the GitHub docs is github/docs@f9b50ac but it has no useful info in the commit message and the linked issue is a 404 so the trail goes cold there.

Caleb Cartwright · Answer 7 · Wed Aug 17 2022 09:07:31 GMT+0800 (China Standard Time)

I think trying to serve information we believe is wrong or outdated first is the wrong call.

Good point. I'd already forgotten that the two endpoints need different inputs

I guess the thing I'd like to be slightly more sure of is: what is the status of the /workflows/<WORKFLOW_NAME> endpoint we are currently using? It is certainly still returning some data. Beyond anec-data: The commit where this changed in the GitHub docs is github/docs@f9b50ac but it has no useful info in the commit message and the linked issue is a 404 so the trail goes cold there.

Yup, I'd found that too and expressed some of the same thinking earlier in the thread. It's an odd scenario and I'm not sure the best way to get in touch with GitHub for a definitive answer on this particular item. Guess we can try the user forum

Renan Pinheiro · Answer 8 · Wed Sep 21 2022 09:11:35 GMT+0800 (China Standard Time)

I have a proposal for this issue:

Add the (legacy) suffix in the name of all the current Github badges with the old path.
Create new badges and new endpoints for the newer Github path, eg. the old path is /github/workflow/status/:user/:repo/:workflow the new can be /github/actions/workflow/status/:user/:repo/:workflowFileName

Images example:

Current behavior:

Proposed new behavior:

I am willing to do the PR too if this proposal eventually is approved 😉

For now, I've created a Vercel app that uses the json badge for any people with this problem:

Example for my repo NextAudio

Shields.io badge:

App badge (using Shields.io json option):

App Repo
App Url

chris48s · Answer 9 · Thu Sep 22 2022 03:40:21 GMT+0800 (China Standard Time)

Tbh, I'd rather just deprecate it rather than keep around a a "legacy" version if we can't rely on the data.

I've started a discussion over at https://github.com/orgs/community/discussions/33739 to try and get an answer. If we don't get to the bottom of it there, I'll try contacting support.

chris48s · Answer 10 · Tue Sep 27 2022 03:00:00 GMT+0800 (China Standard Time)

Nobody replied to https://github.com/orgs/community/discussions/33739
I've now raised a ticket with github support

chris48s · Answer 11 · Fri Sep 30 2022 01:15:00 GMT+0800 (China Standard Time)

I got this message back from support, which is quite helpful and does explain what is happening:

Thank you for your patience while we investigated this.

I have been updated that the file name route is the recommended route and therefore the only one we currently document. The old badge route had some issues disambiguating between workflows with the same name, handling renames, etc. We no longer document that format.

Looking at the first example :

one quirk in both versions of badges is that they first check for matching runs on the default branch (in this case, main) if no branch is provided. If none are found, it will fall back to querying all branches

https://github.com/NextAudio/NextAudio/workflows/.NET/badge.svg searches for a run on the main branch across all workflows named ".NET". It finds https://github.com/NextAudio/NextAudio/actions/runs/1565789342 from 10 months ago. This run looks to be from a now-deleted workflow file build.yml. This is the run whose status is being displayed

https://github.com/NextAudio/NextAudio/actions/workflows/build-test-lint.yml/badge.svg is only querying runs for the specific workflow build-test-lint.yml. There are none on the default branch, so it falls back to all branches and displays the status of https://github.com/NextAudio/NextAudio/actions/runs/3138965027

I think my conclusion from this is that we should:

Create a new badge route that accepts the workflow file name as the param and call /actions/workflows/<WORKFLOW_FILE> to get the data
Deprecate our existing route instead of keeping it around, given it has some known spurious behaviours

I'll submit a PR doing this next week (I'm going to be away for a few days), unless someone wants to jump on it over the weekend.