Every commit now results in a CI failure
msporny opened this issue · comments
@OR13 every new commit results in a CI failure now. This is due to the "Include Breaking" CI flow you added. Everyone watching this repo's inbox is going to be spammed with these messages, which are not helpful. Do you have any suggestions on how to fix this?
https://github.com/w3c/did-test-suite/actions
I agree that having clean output when running one's implementation is very helpful, so we want to keep that. However, getting job failures as the default for every commit to main is not helpful, and over time, will result in us completely ignoring CI failures (which is something you don't want to train people to do).
Were you intending to fix this in a separate PR? Or do we need to disable CI for the full test suite run?
@msporny 2 options:
- stop testing fixtures that break
- remove all fixtures that break
I am happy to do a PR that does either.
It felt wrong to do either in a PR where new stuff was added.
I am happy to do a PR that does either.
Both of those are not good options. One of the goals of a W3C implementation test suite is to see what implementation reality is. Knowing that 30% of the implementations are willfully not conforming to a specific statement in the specification is useful. Those sorts of things sometimes result in specifications being updated to modify or eliminate the normative statement. So, we need to include all fixtures in the test report output, even if they're failing tests.
This is something that is different from the way most CI systems run... in that the CI system has to see no failures for it to report that CI passed. W3C test suites are different, in that we want to report on all the successes as well as all the failures (because at times, the failures are more important to see than the successes).
My suggestion is that we ignore the result of the test suite run when we're doing "Include Breaking" CI. It always succeeds. The only danger there is if it fails for a non-test reason (like fails to run at all). I don't have an answer for that case, but for the time being, we may just want to ignore if there are any errors thrown for specific tests. I don't know if jest has a flag that lets you modify the return code after the program runs. If not, we can do a jest || true.
So, by default, the test suite doesn't run broken test fixtures; this makes it easy for people to add their implementations to the test suite. During CI, the test suite runs broken test fixtures so that they can be included in the report, but ignores the error status of jest. I think that gives us what we need for the next month or two, at least.
@msporny agreed,
- We need the report generation wired into CI
- We make the report operate on all fixtures
- We set the "include broken" status to passing, even when it fails
Its dangerous to turn off those failures, if they are not automatically reflected in a report that is kept up to date... but once the report is automated, I don't see a reason to let CI fail.
@msporny would you like me to do a PR to automate the report? See #146
Yes, please and thank you.
My preference is to use CI to publish the report to the gh-pages branch and use Github pages to expose that on the web, and keep the main branch free of auto-generated files.
Ideally we wouldn't be checking anything auto-generated into Github (and instead publish on an AWS bucket exposed to the web with a w3id.org redirect or something (what we do for the vaxcert test suite)... but that's overkill for what we need right now.
but that's overkill for what we need right now.
Hmm... I wonder if there is a way for us to use ECHIDNA to auto-build/publish the DID Core test suite reports? @iherman -- thoughts?
My guess is probably not... and that ECHIDNA is focused on publishing ReSpec documents... not test suites.
@msporny yes, I like to keep all build / generated assets out of main, I don't have permissions to set the github pages branch though. I can doo everything but that last step.
regarding ECHIDNA, I tested this previously, and it fails to include anything other that the html files... so you need to build / inject them to get it to work... which does not work in, you must modify the index.html before pushing IIRC.. (which is wrong approach IMO).
I suspect there might be a way to make ECHIDNA work gh-pages and automated report building.
but that's overkill for what we need right now.
Hmm... I wonder if there is a way for us to use ECHIDNA to auto-build/publish the DID Core test suite reports? @iherman -- thoughts?
My guess is probably not... and that ECHIDNA is focused on publishing ReSpec documents... not test suites.
Echidna is focused publishing documents onto /TR space... so I don't think that is an option.
Echidna is focused publishing documents onto /TR space... so I don't think that is an option.
NOTEs are published to /TR space and some implementation reports have been published to /TR in the past:
https://www.w3.org/TR/2013/NOTE-prov-implementations-20130430/
https://www.w3.org/TR/2012/NOTE-rdb2rdf-implementations-20120814/
Granted... those links are from almost a decade ago... and were the only two I could find... so the argument is shaky at best. :)
We don't have to publish using ECHIDNA, just wondering if there was a path there that would reduce the amount of work @OR13 would have to do. It sounds like it's not a viable strategy based on what @iherman says above.
Echidna is focused publishing documents onto /TR space... so I don't think that is an option.
NOTEs are published to /TR space and some implementation reports have been published to /TR in the past:
https://www.w3.org/TR/2013/NOTE-prov-implementations-20130430/
https://www.w3.org/TR/2012/NOTE-rdb2rdf-implementations-20120814/Granted... those links are from almost a decade ago... and were the only two I could find... so the argument is shaky at best. :)
We don't have to publish using ECHIDNA, just wondering if there was a path there that would reduce the amount of work @OR13 would have to do. It sounds like it's not a viable strategy based on what @iherman says above.
Indeed (and, incidentally, I was staff contact for both of those cases:-) but that was indeed ages ago, when github was not yet an integral part of the W3C publishing process. AFAIK, these days implementation reports are just kept in github, which also gives more flexibility once the WG is closed.
If we want to go down the ECHIDNA route anyway, here are the admin hooplas to follow
- The WG must pass two resolutions:
- That the implementation report is to be published as a WG Note.
- That the note must be published via ECHIDNA
- Make the first publication by hand, that is:
- Submit a transition request to the director to approve, essentially, the short name (provided the Director is o.k. with this)
- The first version of the note must be published on our CVS server by a team person, ie, yours truly
- The webmaster must add the new note to the official /TR publication database
- I have to generate a github token for the publication
- Set up the local files in the repo to get ECHIDNA working.
We went through these for our notes (eg, use cases), and it all can be done of course. At this moment, however, I do not understand what would be the goal of going through all that, ie, what the problem is ECHIDNA would solve. If the reason is to give a W3C URL to the implementation report, we can simply set up a proxy from /2021/06...
At this moment, however, I do not understand what would be the goal of going through all that, ie, what the problem is ECHIDNA would solve.
The only benefit would be to use the ECHIDNA "auto-publish on commit" machinery instead of having to have @OR13 re-invent it for just this repo. But... he's already done it, so it's a no-op.
It doesn't look like ECHIDNA is a good fit here, happy to drop that option on the floor as we already have something that's working.
This has been resolved, closing.