Run Cucumber Scenarios in Parallel with Maven
- Cucable Maven Plugin
- How it works
- Typical workflow
- Example project
- Appendix
Cucable Maven Plugin
Cucable is a Maven plugin for Cucumber scenarios that simplifies fine-grained and efficient parallel test runs.
This plugin does the following:
- Generate single Cucumber features containing one single scenario each (scenario outlines are also split up into separate scenarios)
- Generating Cucumber runners
- for every generated "single scenario" feature file or
- for multiple generated "single scenario" feature files
Those generated runners and features can then be used with Maven Failsafe in order to parallelize test runs.
Note: From version 0.1.7 on this also works for non-english feature files!
Repository Structure
- plugin-code contains the full plugin source code.
- example-project contains an example Maven project to see the plugin in action.
Changelog
All changes are documented in the full changelog.
Maven dependency
<dependency>
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>(check version on top of the page)</version>
</dependency>
How it works
- Cucable will cut up feature file into the smallest possible runnable scenarios
- Each generated feature file includes a single scenario
- After this, the runner classes for those generated features are generated based on a provided template file, either
- one runner per generated "single scenario" feature file or
- one runner per group of "single scenario" feature files
Template placeholders
[CUCABLE:RUNNER]
The [CUCABLE:RUNNER]
template placeholder is automatically replaced with the class name of the generated runner class.
If the generated runner runs only one "single scenario" feature, its name will be the same as the generated feature (e.g. MyFeature_scenario001_run001_IT
).
In case the runner runs multiple "single scenario" features, its name will be auto-generated (e.g. CucableMultiRunner_1da810a2_c4c6_4edb_b078_d81329593950_IT
).
[CUCABLE:FEATURE]
The [CUCABLE:FEATURE]
can be placed in the feature
option of the @CucumberOptions
block in your template:
@CucumberOptions( features = {"target/parallel/features/[CUCABLE:FEATURE].feature"} )
Cucable will automatically detect the string containing the [CUCABLE:FEATURE]
placeholder and use this to generate one line for each feature this runner should trigger.
Custom template placeholders - [CUCABLE:CUSTOM:xxx]
In some cases, you may need to set custom values that should be written to your template files.
In this case, just add a block to your POM file:
<customPlaceholders>
<somename>Any value</somename>
<foo>bar</foo>
</customPlaceholders>
These custom placeholders can be used anywhere in your template:
import cucumber.api.CucumberOptions; @CucumberOptions( features = {"target/parallel/features/[CUCABLE:FEATURE].feature"}, plugin = {"json:target/cucumber-report/[CUCABLE:CUSTOM:foo].json"} ) public class [CUCABLE:RUNNER] { // [CUCABLE:CUSTOM:somename] }
In this case the result would be
import cucumber.api.CucumberOptions; @CucumberOptions( features = {"target/parallel/features/[CUCABLE:FEATURE].feature"}, plugin = {"json:target/cucumber-report/bar.json"} ) public class [CUCABLE:RUNNER] { // Any value }
Note: The custom placeholder names are case sensitive!
One runner per generated scenario
This is the default mode of Cucable. Having multiple runners that run one "single scenario" feature each is best for parallelization with Maven Failsafe.
One runner per group of generated scenarios
If you set the desiredNumberOfRunners
option to a number greater than 0, Cucable will automatically switch to the multi-feature runner mode.
This means that it will only generate the specified number of runners and distribute the generated features equally to each one of them. This is helpful if a group of scenarios should be executed during each forked run of your test framework.
Note: If a runner runs only one feature, it automatically has the same name as the feature. Otherwise it will have a unique auto-generated name.
Typical workflow
- Generation of runners and features
- Running the generated tests with Maven Failsafe
- Aggregation of a single test report after all test runs
The following sections break down the above steps.
1. Generation of runners and features
<plugin>
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>${cucable-plugin.version}</version>
<executions>
<execution>
<id>generate-test-resources</id>
<phase>generate-test-resources</phase>
<goals>
<goal>parallel</goal>
</goals>
</execution>
</executions>
<configuration>
<!-- Required properties -->
<sourceRunnerTemplateFile>src/test/resources/parallel/cucable.template</sourceRunnerTemplateFile>
<sourceFeatures>src/test/resources/features</sourceFeatures>
<generatedFeatureDirectory>src/test/resources/parallel/features</generatedFeatureDirectory>
<generatedRunnerDirectory>src/test/java/parallel/runners</generatedRunnerDirectory>
<!-- Optional properties -->
<numberOfTestRuns>1</numberOfTestRuns>
<includeScenarioTags>
<param>@includeMe</param>
<param>@includeMeAsWell</param>
</includeScenarioTags>
<excludeScenarioTags>
<param>@skip</param>
</excludeScenarioTags>
<logLevel>compact</logLevel>
<desiredNumberOfRunners>2</desiredNumberOfRunners>
</configuration>
</plugin>
Required Parameters
sourceRunnerTemplateFile
The specified file will be used to generate runner classes for the generated feature file that can be run using Maven Failsafe.
This can be either a text file or a Java class. The difference can be seen below:
Using a java file as a runner template
If you use a java file (e.g. src/test/java/some/template/CucableJavaTemplate.java), the [CUCABLE:FEATURE] placeholder as well as the class name will be substituted for the generated feature file name(s). The [CUCABLE:RUNNER] placeholder will be replaced by the runner class name.
Additionally, the package declaration will be stripped.
Example:
package some.template; import cucumber.api.CucumberOptions; @CucumberOptions( features = {"target/parallel/features/[CUCABLE:FEATURE].feature"}, plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"} ) public class CucableJavaTemplate { }
will turn into
import cucumber.api.CucumberOptions; @CucumberOptions( features = {"target/parallel/features/MyFeature_scenario001_run001_IT.feature"}, plugin = {"json:target/cucumber-report/MyFeature_scenario001_run001_IT.json"} ) public class MyFeature_scenario001_run001_IT { } // Generated by Cucable from src/test/java/some/template/CucableJavaTemplate.java
In case of a fixed number of runners that contain multiple scenarios (via desiredNumberOfRunners
property), the runner name will be auto-generated:
import cucumber.api.junit.Cucumber; import cucumber.api.CucumberOptions; import org.junit.runner.RunWith; @RunWith(Cucumber.class) @CucumberOptions( features = {"target/parallel/features/Bookmarks_scenario001_run001_IT.feature", "target/parallel/features/Bookmarks_scenario003_run001_IT.feature"}, plugin = { "json:target/cucumber-report/CucableMultiRunner_1da810a2_c4c6_4edb_b078_d81329593950_IT.json", "com.trivago.trupi.plugin.TrupiCucumberPlugin" }, glue = {"com.trivago.trupi.glue"} ) public class CucableMultiRunner_1da810a2_c4c6_4edb_b078_d81329593950_IT { } // Generated by Cucable from src/test/java/some/template/CucableJavaTemplate.java
Using a text file as a runner template
If you use a text file (e.g. src/test/resources/cucable.template), all [CUCABLE:FEATURE] placeholder will be substituted for the generated feature file name(s). The [CUCABLE:RUNNER] placeholder will be replaced by the runner class name.
sourceFeatures
This can specify
- the root path of your existing Cucumber .feature files (e.g.
src/test/resources/features
) - the path to a specific existing Cucumber .feature file (e.g.
src/test/resources/features/MyFeature.feature
) - the path to a specific existing Cucumber .feature file including line numbers of specific scenarios/scenario outlines inside this file (e.g.
src/test/resources/features/MyFeature.feature:12:19
would only convert the scenarios starting at line 12 and 19 inside MyFeature.feature)
generatedFeatureDirectory
The path where the generated Cucumber .feature files should be located (e.g. src/test/resources/parallel).
Note: This directory should be located under a valid resource folder to be included as a test source by Maven. If you want to use a directory inside Maven's target folder, check this example.
Caution: This directory will be wiped prior to the feature file generation!
generatedRunnerDirectory
The path where the generated runner classes should be located (e.g. src/test/java/parallel/runners).
Note: This directory should be located under a valid source folder to be included as a test source by Maven. If you want to use a directory inside Maven's target folder, check this example.
Caution: This directory will be wiped prior to the runner file generation!
Optional Parameters
numberOfTestRuns
Optional number of test runs. This can be used if specific scenarios should be run multiple times. If this options is not set, its default value is 1.
For each test run, the whole set of features and runners is generated like this:
- MyFeature_scenario001_run001_IT.feature
- MyFeature_scenario001_run002_IT.feature
- MyFeature_scenario001_run003_IT.feature
- etc.
Note: Characters other than letters from A to Z, numbers and underscores will be stripped out of the feature file name.
includeScenarioTags
Optional scenario tags that should be included in the feature and runner generation.
To include multiple tags, just add each one into as its own <param>
:
<includeScenarioTags>
<param>@scenario1Tag1</param>
<param>@scenario1Tag2</param>
</includeScenarioTags>
Note: When using includeScenarioTags and excludeScenarioTags together, the excludeScenarioTags will override the includeScenarioTags. This means that a scenario containing an included tag and an excluded tag will be excluded!
excludeScenarioTags
Optional scenario tags that should not be included in the feature and runner generation.
To include multiple tags, just add each one into as its own <param>
:
<excludeScenarioTags>
<param>@tag1</param>
<param>@tag2</param>
</excludeScenarioTags>
Note: When using includeScenarioTags and excludeScenarioTags together, the excludeScenarioTags will override the includeScenarioTags. This means that a scenario containing an included tag and an excluded tag will be excluded!
logLevel
By default, Cucable logs all information including
- its own name and version
- all passed property values
- a list of processed feature paths
This can be configured by passing the logLevel
property:
<logLevel>default|compact|minimal|off</logLevel>
- default will log all the mentioned information
- compact will only log the plugin name, version, properties and one line of summary
- minimal will only log a summary line
- off will prevent any logging
desiredNumberOfRunners
If you set this options, all generated features will be distributed to a fixed set of runner classes. This means that one runner can potentially run multiple features in sequence.
If this option is not set, its default value is 0
which basically means "Generate a dedicated runner for every generated feature".
Generating runners and features inside target directory
It may be desirable for you to generate the Cucable features and runners in Maven's target
directory.
The advantage of this is that this directory is wiped by the mvn clean
command and older generated files do not reside in your src
directory.
In order to achieve this, you can specify subdirectories under target (${project.build.directory}
) for Cucable, e.g. ${project.build.directory}/parallel/runners
and ${project.build.directory}/parallel/features
After this step, use the build-helper-maven-plugin in your POM file in order to consider the generated runner classes test sources:
<plugins>
<plugin>
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>${cucable.plugin.version}</version>
<executions>
<execution>
<id>generate-test-resources</id>
<phase>generate-test-resources</phase>
<goals>
<goal>parallel</goal>
</goals>
</execution>
</executions>
<configuration>
<sourceRunnerTemplateFile>path_to_template_file</sourceRunnerTemplateFile>
<sourceFeatures>path_to_features</sourceFeatures>
<generatedFeatureDirectory>${project.build.directory}/parallel/features</generatedFeatureDirectory>
<generatedRunnerDirectory>${project.build.directory}/parallel/runners</generatedRunnerDirectory>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>${build.helper.plugin.version}</version>
<executions>
<execution>
<id>add-test-source</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
</execution>
</executions>
<configuration>
<sources>
<source>${project.build.directory}/parallel/runners</source>
</sources>
</configuration>
</plugin>
</plugins>
Complete Example
Below, you can see a full example of what Cucable does.
Source feature file
This is our source feature file. It contains a scenario and a scenario outline with two examples.
MyFeature.feature
Feature: This is the feature name
Scenario: First scenario
Given I am on the start page
And I click the login button
Then I see an error message
Scenario Outline: Second scenario with an amount of <amount>
Given I am on the start page
And I add <amount> items
And I navigate to the shopping basket
Then I see <amount> items
Examples:
| amount |
| 12 |
| 85 |
Runner template file
This is the runner template file (in this example we use a text file) that is used to generate single scenario runners.
- The placeholder [CUCABLE:FEATURE] and its enclosing string will be replaced with the generated feature names by Cucable.
- The placeholder [CUCABLE:RUNNER] will be replaced with the generated runner class name by Cucable.
import cucumber.api.CucumberOptions; @CucumberOptions( features = {"target/parallel/features/[CUCABLE:FEATURE].feature"}, plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"} ) public class [CUCABLE:RUNNER] { }
Note: The specified plugin generates Cucumber JSON files which are needed for custom aggregated test reports.
Generated Scenarios
For each scenario, a single feature file is created:
MyFeature_scenario001_run001_IT.feature
Feature: This is the feature name
Scenario: First scenario
Given I am on the start page
And I click the login button
Then I see an error message
Note that for the scenario outlines, each example is converted to its own scenario and feature file:
MyFeature_scenario002_run001_IT.feature
Feature: This is the feature name Scenario: Second scenario with an amount of 12 Given I am on the start page And I add 12 items And I navigate to the shopping basket Then I see 12 items
MyFeature_scenario003_run001_IT.feature
Feature: This is the feature name Scenario: Second scenario with an amount of 85 Given I am on the start page And I add 85 items And I navigate to the shopping basket Then I see 85 items
Generated runners
The generated runners point to each one of the generated feature files (unless you use the desiredNumberOfRunners
option).
This is an example for one of the generated runners - note how the placeholders are now replaced with the name of the feature to run:
MyFeature_scenario001_run001_IT.java
import cucumber.api.CucumberOptions; @CucumberOptions( features = {"target/parallel/features/MyFeature_scenario001_run001_IT.feature"}, plugin = {"json:target/cucumber-report/MyFeature_scenario001_run001_IT.json"} ) public class MyFeature_scenario001_run001_IT { }
2. Running the generated tests with Maven failsafe
This will skip the unit tests (if any) and run the generated runner classes with Maven Failsafe. Since all generated runner classes from the step before end with _IT, they are automatically considered integration tests and run with Maven Failsafe.
Note: If all tests should be run regardless of their result, it is important to set <testFailureIgnore>true</testFailureIgnore>
for Maven Failsafe - otherwise the plugin execution will stop on failing tests.
However, if this is specified, the build will not fail in case of failing tests!
To circumvent that, it is possible to specify a custom rule for Maven enforcer that passes or fails the build depending on custom conditions.
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>Run parallel tests</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<forkCount>${maven.fork.count}</forkCount>
<reuseForks>false</reuseForks>
<argLine>-Dfile.encoding=UTF-8</argLine>
<disableXmlReport>true</disableXmlReport>
</configuration>
</plugin>
</plugins>
3. Aggregation of a single test report after all test runs
We use the Cluecumber plugin to aggregate all generated .json report files into one overall test report.
<plugins>
<plugin>
<groupId>com.trivago.rta</groupId>
<artifactId>cluecumber-report-plugin</artifactId>
<version>${cluecumber.report.version}</version>
<executions>
<execution>
<id>report</id>
<phase>post-integration-test</phase>
<goals>
<goal>reporting</goal>
</goals>
</execution>
</executions>
<configuration>
<sourceJsonReportDirectory>${project.build.directory}/cucumber-report</sourceJsonReportDirectory>
<generatedHtmlReportDirectory>${project.build.directory}/test-report</generatedHtmlReportDirectory>
</configuration>
</plugin>
</plugins>
Example project
You can test the complete flow and POM configuration by checking out the Cucable example project.
Appendix
Building
Cucable requires Java >= 8 and Maven >= 3.3.9. It is available in Maven central.
License
Copyright 2017 trivago N.V.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Credits
This plugin was inspired by the Cucumber Slices Maven Plugin.