Cannot execute scenarios in parallel
mmitteregger opened this issue · comments
Executing multiple scenarios from the same feature in parallel does not work due to exceptions.
java.lang.NullPointerException: No BaseStepListener has been registered
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:787)
at net.thucydides.core.steps.StepEventBus.getBaseStepListener(StepEventBus.java:131)
at net.thucydides.core.steps.StepEventBus.addDescriptionToCurrentTest(StepEventBus.java:600)
at cucumber.runtime.formatter.SerenityReporter.startScenario(SerenityReporter.java:556)
at cucumber.runtime.formatter.SerenityReporter.startOfScenarioLifeCycle(SerenityReporter.java:547)
at cucumber.runtime.formatter.SerenityReporter.handleTestCaseStarted(SerenityReporter.java:219)
at cucumber.runtime.formatter.SerenityReporter.lambda$new$1(SerenityReporter.java:112)
at cucumber.runner.EventBus.send(EventBus.java:28)
at cucumber.api.TestCase.run(TestCase.java:55)
at cucumber.runner.Runner.runPickle(Runner.java:80)
at cucumber.runtime.junit.PickleRunners$NoStepDescriptions.run(PickleRunners.java:140)
...
You have mentioned parallel execution of the scenarios in https://johnfergusonsmart.com/running-parallel-tests-serenity-bdd-cucumber/, but only talked about broken reporting due to scenario outlines.
I was thinking about adding a new parallelScheme to the cucumber-jvm-parallel-plugin plugin, that works like SCENARIO but keeps the Scenario Outlines for Serenity but these issues would need to be resolved first.
Additional details:
- Versions:
<serenity-core.version>1.9.17</serenity-core.version>
<serenity-cucumber.version>1.9.7</serenity-cucumber.version>
<serenity-junit.version>1.9.17</serenity-junit.version>
<serenity-rest-assured.version>1.9.17</serenity-rest-assured.version>- Operating system: Linux
- Example project: https://github.com/mmitteregger/bugreport-serenity-core-parallel-scenarios
This is not supported at th moment (most likely the new threads created when scenarios are run do not call all the Cucumber lifecycle events, so the reporting support is not present in these threads. Feel free to take a look if you can see what is going on.
I don't think this is doable with the current Cucumber and cucumber-jvm-parallel-plugin implementations.
Is there any Alternative?
What would make it possible?
No. It might be possible with Cucumber 4, as they have a way of ensuring the reporting events are sent in sequence even if they are run in parallel, but we are still figuring out how to upgrade to Cucumber 4 without forcing everyone to rewrite all there test suites (Cucumber 4 is not backward compatible with Cucumber 2.x).
I create custom code to make it parallel. So I use maven invoker to do that. it is like running mvn clean verify with different parameter. My solution is:
- Read the all the feature file and convert it into AST Model (See gherkin documentation to do this)
- Then filter which parameter that need to run by comparing the tags. you need TagExpressionParser class from cucumber depedency
- Allocate for each scenario evenly for each thread.
- Run it using maven invoker. you can use ExecutorService class.
@Hendrione Running the scenarios in parallel is not so much the problem, it's accounting for the scenarios in the reports, in particular if each row of an example table is run as a separate scenarios (if example tables are run as a single block, it would work, but the Cucumber model doesn't make this distinction).
ya. the problem is the report right, so for right now we create custom reports for it. And still, it is connected to the Serenity report. This is our team solution right now to do parallel test.
Does Serenity support parallelization using Cucumber 4+ now?
No, we are still working on this.
Is this fixed?
No.
Any workaround?
You can have separate runner classes and run them in parallel.