How will be produce visualization of the test results
OR13 opened this issue · comments
- At minimum we want to see Green checkmarks Red X's for tests in the spec.
- List fo implementers for every test.
- Goal is in each row to have at least 2 greens.
- Having a human readable description of what each test does it a nice to have.
- Also good to have a few words on each implementation
We should hold off on this until we feel the suite is very stable.
But we can gather snippets and examples here in the meantime
We should also make sure we take advantage of automated visualization, since this is free with JEST.
Another example from the WG call:
- https://w3c.github.io/publ-tests/test_reports/manifest_processing/index.html
- built from https://github.com/w3c/publ-tests/blob/main/publication_manifest/manifest_processing/reports/colibrio-reader-framework.json
With separate meta data for the test suite:
I have started to try to structure the test output in a more hierarchical, logical fashion in PR #52. The general structure is: major section of the spec > implementation being tested > minor section of the spec > specific variation of implementation being tested > specific normative statement. For example:
- 3.x Identifier
- id:key (2018 cryptosuite) - did-method-key-js - Digital Bazaar
- 3.1 DID Syntax
did:key:zMk623kjsd...8fd- 3.1 DID Syntax - MUST be a valid URL.
- 3.1 DID Syntax
- id:key (2018 cryptosuite) - did-method-key-js - Digital Bazaar
It's a bit awkward, but given that test suite implementers have chosen to test different sections of the spec in different ways, it's the only logical structuring that we can do, AFAICT.
The issue was discussed in a meeting on 2021-04-06
- no resolutions were taken
View the transcript
7.2. How will be produce visualization of the test results
See github issue did-test-suite#51.
Markus Sabadello: I'm not familiar with this one, self describing
… test report in the form of a json file
… a number o fways of how to render that in html
Daniel Burnett: even though we don't have a notion of editorial with the test suite, this falls into that category
… presentation, not about the actual running of the tests
… in the end orie will decide something but if you have an opinion, express it
Ivan Herman: we discussed this with orie on the last call
… we will have to produce some sort of report of the whole CR phase, test results, implementations, etc
… there are other examples in other WGs
… I can help with that at the end
… the only thing it affects right now is that there should be a clear structured way of characterising each test
… metadata about the test
… and about the implementations
… once those are there in some format, whatever format, we can do something at the end
This has been implemented by @shigeya, closing.