Properties for test cases
dennisgrck opened this issue · comments
Hello there,
I would like to suggest adding support for properties on a test case level. More and more CI and test management tools support this, and this is useful to include & report additional fields and properties for tests (such as test descriptions, test attributes, test steps details etc). This is also supported by various other test frameworks such as pytest or Playwright, as well as reporting and test management tools such as Testmo.
Properties for test cases
Here is an example of what test case properties look like in the resulting XML file. This is the same format as supported for testsuite element, just below the testcase element:
<testcase name="testCase3" classname="Tests.Registration" time="3.441">
<properties>
<property name="priority" value="high" />
<property name="language" value="english" />
<property name="author" value="Adrian" />
<property name="attachment" value="screenshots/dashboard.png" />
<property name="attachment" value="screenshots/users.png" />
<!-- Optional support for properties with text values -->
<property name="description">
This text describes the purpose of this test case and provides
an overview of what the test does and how it works.
</property>
</properties>
<!-- Optional output or error/failure/skipped -->
</testcase>You can also find additional details on this format here as well as a full XML example file.
Possible API suggestion
Similar to how the Mocha JUnit reporter already supports this.test.consoleOutputs and this.test.attachments, I would suggest adding a new optional this.test.properties field. This could be an array of objects with name/value fields. I would suggest this API as it allows the use of multiple properties with the same name, as this is useful to include e.g. multiple steps, attachments etc. Here's what a possible API could look like:
this.test.properties = [
{ name: 'author', value: 'Adrian' },
{ name: 'priority', value: 'high' },
{ name: 'attachment', value: 'screenshots/dashboard.png' },
{ name: 'attachment', value: 'screenshots/users.png' },
{ name: 'attachment', value: 'screenshots/error.png' },
{ name: 'description, value: 'Multi-line properties with text values..', text: true },
];I'm happy to provide more details on how other tools are using these properties. Thanks for considering this feature request!
@dennisgrck , fyi, the links for the documentation you provided are not working.
@bitcoder Thanks a lot for letting me know, I've fixed the links now!
@michaelleeallen @clayreimann I would be happy to implement this and prepare a PR if you think this would be interesting to review. I would make sure this is a completely tool-agnostic implementation and follows the conventions of other tools that already support this (such as pytest and various other frameworks).
IMO a tool-agnostic way to include test case properties in the resulting XML file will allow many developers to build better integrations with Mocha & various reporting, CI & testing tools that support test case properties already.
How would that fit exactly a mocha test and a Cypress test for example?
The proposed API would work exactly like the existing APIs such as this.test.attachments and this.test.consoleOutputs, so it would fit well with Mocha and Cypress like the existing approach. Hope this clarifies it!
Well, not totally :)
How would you specify those right from the specification block (i.e., it/describe)? Not from within it. Because some metadata must be specified before and not during the test
@bitcoder This is not the intention of this issue request and not in the scope of what I suggested. My suggestion (and possible PR) is to provide a simple API to specify properties, just like the existing attachment & output APIs. I don't think it's a good idea to extend the function signature for this and this is also not really necessary for my suggestion; any property that can be specified in the function signature can also be specified within the test instead, which works just like the existing APIs. Hope this clarifies it!
To add some additional information to this request, other popular testing tools such as Pytest and Playwright also support test case properties the same way as the above suggestion, and they also support multiple properties with the same name. Here is an example for Playwright (which uses Playwright's annotations API):
test('user profile', async ({ page }) => {
test.info().annotations.push({ type: 'severity', description: 'critical' });
test.info().annotations.push({ type: 'language', description: 'english' });
// Playwright also supports multiple properties with the same name
test.info().annotations.push({ type: 'step', description: 'The first step' });
test.info().annotations.push({ type: 'step', description: 'The second step' });
test.info().annotations.push({ type: 'step', description: 'The third step' });
// [..]
});And here is the same example for Python & Pytest:
def test_function(record_property):
record_property("severity", "critical")
record_property("language", "english")
# Pytest also supports multiple properties with the same name
record_property("step", "The first step")
record_property("step", "The second step")
record_property("step", "The third step")
assert TrueInstead of using a tool-specific implementation like #153 from one vendor, I would like to suggest using a similar open API for this project. The mocha-junit-reporter package is used with many frameworks and tools, and it is difficult to change APIs later, so I think it would be important to implement this in a more generic way so users can use this with any tool in the future.
@dennisgrck Sorry for the delay, I'd be happy to review a PR if you were to implement this feature. As long as nothing breaks most any contribution is welcome here
Sounds good, I will work on this and will try to get the PR ready in the next days.