Test Failures despite appropriate response

Question

Test Failures despite appropriate response

mkhraisha opened this issue 2 years ago · comments

Mahmoud Alkhraishi commented 2 years ago

We have ran into a couple of failing tests where when reconstructed manually via postman they return the correct response however in this test suite they are failing.
This applies to:

credential.credentialSubject should not be empty
"credential.issuer" MUST be a string or an object

Our implementation returns a 400, with a body that outlines what is missing, and the test suite marks that as a failure.

Manu Sporny · Answer 1 · Fri Nov 04 2022 05:18:32 GMT+0800 (China Standard Time)

Hey @aljones15 thoughts on the above?

Andrew Jones · Answer 2 · Fri Nov 04 2022 23:23:59 GMT+0800 (China Standard Time)

I will look into this, but I checked here: https://w3c-ccg.github.io/vc-api-issuer-test-suite/

And for both test it says mavenet is returning a response and not an error. It might be the case that it is returning a response with a status 400 and the issue is on our side. I won't be able to check this today though as I am not at home and I keep the oauth2 stuff on my computer at home. @mkhraisha thanks for filing the issue and I should be able to look at this soon.

Andrew Jones · Answer 3 · Sun Nov 06 2022 03:14:05 GMT+0800 (China Standard Time)

@mkhraisha ok so I don't have the latest oauth2 token so I can't actually test the credentialSubject locally to debug, but some of our tests are in fact compound tests where multiple invalid states are tried that all sum up to a single normative statement. This PR changes that for the credentialSubject tests, by breaking them out into single negative tests: #34

Andrew Jones · Answer 4 · Sun Nov 06 2022 03:30:36 GMT+0800 (China Standard Time)

@mkhraisha the issuer tests have also been broken out into several new negative and positive tests here: #35

Manu Sporny · Answer 5 · Sun Nov 06 2022 08:50:07 GMT+0800 (China Standard Time)

@mkhraisha the issuer tests have also been broken out into several new negative and positive tests here: #35

We can't do this as it makes it seem as if normative requirements are being asserted that are not in any spec. This will cause people to get confused about the normative statements in the spec. I left some suggestions in the PR.

Andrew Jones · Answer 6 · Mon Nov 07 2022 22:59:21 GMT+0800 (China Standard Time)

@mkhraisha ok so have some actual results:

"credential.credentialSubject" MUST NOT be null - passing (Mavenet returns 400 HTTPError with a really good message)
"credential.credentialSubject" MUST NOT be a boolean - failing mavenent returns 201
"credential.credentialSubject" MUST NOT be a number - failing mavenent returns 201
"credential.credentialSubject" MUST NOT be an empty Array - failing mavennet returns 201
"credential.credentialSubject" MUST NOT be a string - failing mavennet returns 500

I don't have them for the credential.issuer tests yet. Thank you for raising this issue again as it has clarified the test results quite a bit.

Mahmoud Alkhraishi · Answer 7 · Mon Nov 07 2022 23:09:32 GMT+0800 (China Standard Time)

Thank you!

This helps quite a bit. Will dig into these on my end.