Simplify test identification in the SauceLabs dashboard when concurrency mode is enabled
amidongarrett opened this issue · comments
We are trying to run our test suite (2 tests) concurrently on Saucelabs. However, when they finish, testcafe does not report to Saucelabs correctly if the test passed/failed, resulting in Saucelabs timing out and marking each test as failed.
Since the naming of the tests are done through a single environment variable, all of our tests that run will have the same name, making it impossible to differentiate between tests on Saucelabs. Without having some unique identifier that we provide the Saucelabs metadata, we cannot use any of their APIs to update the tests in an after/afterEach hook.
Is it possible (whether in the Runner class or test) to provide users with the Job IDs or to make it so the naming of tests in done by an environment variable and (if not provided) fallback to the test name?
I am currently running in this situation. Is it possible to do it ?
Having the same trouble. Im running concurrently 3 tests and im not able to modify the env var in node fast enough for each test to run with a different job name
In the concurrency mode, TestCafe creates identical browsers instances and dispatches tests to each instance on demand. It means that we can't predict which test will be run on which browser. That's why I think that assigning different job IDs for different browsers in the concurrency mode is not very useful.
I feel that we can implement the following enhancements to simplify identifications of failed tests in the SauceLabs dashboard:
- Currently, TestCafe reports the total test status of all tests to each SauceLabs job. Instead, TestCafe should report different total status for different jobs, and each total status should take into account only tests dispatched to each particular job.
- When a test fails, TestCafe should include a link to a corresponding SauceLabs job into the error message.
This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.
We're closing this issue after a prolonged period of inactivity. If it still affects you, please add a comment to this issue with up-to-date information. Thank you.