cqframework / cql-tests-runner

Test Runner for CQL Tests

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add Support for Expecting an Error

evan-gordon opened this issue · comments

We would like to be able to formally expect errors in tests. This would help formalizing cases where an engine should fail to execute. Today there are tests with comment tags that say something to the effect of "this should be an error", making this change would formalize these cases.

Option One:

Add an tag for tests where we expect the test to return an error

Option Two:

Place tests that we expect to return error in a different location.

Out of scope:

We could have a discussion on formalizing error representations in CQL engines but for the purposes of this issue we should just worry about whether or not a test should return an error.

The current test specification supports this, and the runner takes advantage of this capability, see the evaluation of test results here:

https://github.com/cqframework/cql-tests-runner/blob/main/cql-tests-runner.js#L65

and here:

https://github.com/cqframework/cql-tests-runner/blob/main/cql-tests-runner.js#L185

Note that one of the topics we discussed at the WGM was creating error codes for the errors so that we can test error codes, rather than testing that specific messages are coming out, but that's a different issue.