Report generation is brittle and needs some development work
brentzundel opened this issue · comments
We have need for additional developer resources to help finish the automatic generation of test implementation report.
Much work has been done:
https://github.com/w3c/did-test-suite/blob/main/packages/did-core-test-server/package.json#L11https://github.com/w3c/did-test-suite/tree/main/packages/did-core-test-server#usagehttps://github.com/w3c/did-test-suite/blob/main/packages/did-core-test-server/generate-latest-respec-data.js
https://github.com/w3c/did-test-suite/blob/main/index.html#L686
Please comment here if you may be able to help, and reach out to @OR13 for more detail.
Have you tried a custom reporter? I've written two in Mocha and the last one has actually turned out to be pretty useful and painless to use.
The later produces the nightly reports for the vaccine credentials interoperability report.
https://transmute-industries.github.io/did-core/
https://github.com/transmute-industries/did-core/blob/main/packages/did-core-tests/jest.config.js#L6
But these don't solve the problem of an NxN breakdown where you can count feature support....
Luckily this exists:
https://github.com/digitalbazaar/vaccination-certificate-test-suite
http://vaxcert-interop-reports.s3-website.us-east-2.amazonaws.com/#COVID-19
^ maybe you can add some support for something like this? Built on the existing JSON which the test suite produces?
Here is the current json: https://github.com/w3c/did-test-suite/blob/main/test-vectors/did-spec.latest.json
^ this is automatically produced by jest.
@OR13 I still don't understand how I can generate a report. How can I configure ./suites/did-spec/default.json ? Maybe, there is something missing in my knowledge...
This line updates the latest json for the report in the html:
This line loads the json from the html which is describes the latests run:
https://github.com/w3c/did-test-suite/blob/main/index.html#L869
Looks like I forgot that this PR never got merged....
@msporny @peacekeeper ^ perhaps this would have helped with report rendering.
Looks like I forgot that this PR never got merged....
https://github.com/w3c/did-test-suite/pull/20/files#diff-9893d200a81ade372ac94c51de4364d040a15fa75250a8b8be6a742d6d4e0697R2
This looks like what I want to know for a long time :)
Now I getting understand the situation. I may help improve the report generation.
@OR13
I want to create a PR based on the closed PR #20 you mentioned above.
Now, I found GitHub doesn't allow me to fetch commits from a closed PR branch.
Would you please kindly reopen PR #20 for this purpose?
Or, if there is an alternative way to receive the commits in PR #20 without reopening it, that's fine for me too.
hmm I can't seem to reopen it.
hmm I can't seem to reopen it.
hmmm. then, can you send the branch to my repository as a new PR?
I can open a PR to your repo, but it won't be mergeable.
The PR in question is so old that the content will need to be reimplemented.
Does this help: https://github.com/w3c/did-test-suite/pull/103/files ?
My git commits are not super clean, low chance you can cherry pick them...I don't care about attribution to me at all, feel free to copy the content that is needed.
I can open a PR to your repo, but it won't be mergeable.
I understand that.
The PR in question is so old that the content will need to be reimplemented.
Does this help: https://github.com/w3c/did-test-suite/pull/103/files ?
Yes. I could fetch files locally. Thanks.
My git commits are not super clean, low chance you can cherry pick them...I don't care about attribution to me at all, feel free to copy the content that is needed.
Thank you.
https://transmute-industries.github.io/did-core/
https://github.com/transmute-industries/did-core/blob/main/packages/did-core-tests/jest.config.js#L6But these don't solve the problem of an NxN breakdown where you can count feature support....
Luckily this exists:
https://github.com/digitalbazaar/vaccination-certificate-test-suite
http://vaxcert-interop-reports.s3-website.us-east-2.amazonaws.com/#COVID-19^ maybe you can add some support for something like this? Built on the existing JSON which the test suite produces?
Here is the current json: https://github.com/w3c/did-test-suite/blob/main/test-vectors/did-spec.latest.json
^ this is automatically produced by jest.
support for the interoperability matrices in that vaccine credentials report is in that w3c-interop-reporter.
This is the template for the matrix reports:
https://github.com/digitalbazaar/vaccination-certificate-test-suite/blob/main/matrix.hbs
The usage section of the readme should explain how to add the data so it shows up in the report:
https://github.com/digitalbazaar/mocha-w3c-interop-reporter
alternatively you can see the actual code that adds the metadata used by that template here: https://github.com/digitalbazaar/vaccination-certificate-test-suite/blob/main/tests/10-interop.js
As for using the json outputed by jest that is completely possible actually. The test project for the reporter uses two mocha JSON test results that are passed to a function called makeReport
https://github.com/digitalbazaar/mocha-w3c-interop-reporter/blob/main/test/mocha/10-render.js
however makeReport is not part of the public API of that reporter sadly. It would not be that hard to make that API exposed if someone wanted to expose it somehow (a binary/command line option might be a good idea).
I think PR #105 (currently in Draft) now has enough code to create a matrix.
Question is, how we want to aggregate into a matrix.
Also, we may want a section with an aggregated result that has a table on numbers related to the CR-exit condition.
Now, PR #105 is ready. Please review.
Thanks to @shigeya , report generation works really well now. Closing.