Proposal of the test set descriptor update

[László Langó]

I'd like to propose an update for testsets.json. You sad before that not everyone happy with current situation with extra testsets files which can update the default settings. That modification was necessary to test different configurations. This is a proposal for the expected testsets.jons, if we want to use one single json descriptor file again (before #1341). With this modification we can keep the ability to test different configurations with a single json.

{
  "directory": [
    { "name": "filename",
      "skip": ["all"],
      "reason": "reason of skipping",
      "timeout": seconds,
      "expected-failure": true,
      "required-modules": ["my_module"],
      "required-features": ["es-262-feature"]
    },
    ...
  ]
}

• directory: group of tests
• name: filename = testname
• skip: platform where the test must be skipped. ["all", "darwin", "linux", "nuttx", "tizen", "tizenrt"] (optional)
• reason: it belongs to skip property, reason of skipping. (optional)
• timeout: timeout in seconds (optional)
• expected-failure: identifies the "must fail" testcases. Still catches segfaults, IOTJS_ASSERT and JERRY_ASSERT. Default: false [true, false] (optional)
• required-modules: List of the required modules. Skips test if the module is not present int the current build. (optional)
• required-features: List of the required language features. eg.: Promise (optional)

Note:

• "required-modules" and "required-features" are compared with the output of tools/iotjs_build_info.js which is currently used by the existing testrunners. It must be extended with the language related features, like Promis, ArrayBuffer, etc.

What do you think?

CC: @daeyeon, @hs0225, @galpeter

[HoSung Kim]

@LaszloLango I agree with your proposal. I look better to combine the test set file into one.

[Roland Takacs]

I like this proposal. Especially, that every tests could have information about their used modules.