Automation with Cucumber JVM
Compatibility with Cucumber JVM 5 or greater
If your project uses a version 4 of Cucumber JVM, select the Cucumber 4 technology. Otherwise, if it uses Cucumber JVM 5 or posterior, select the Cucumber 5+ technology.
Test reference in Squash TM
Depending on the report test precision you wish to apply to your test case, you can refine it down to a feature (in the case your .feature file contains more than one feature, which is not recommended even though Cucumber allows it) or to a specific scenario.
In order to bind a Squash TM test case to a Cucumber automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[repository]/[project]#[feature_file]#[feature_name]#[scenario]
or
[repository]#[feature_file]#[feature_name]#[scenario]
(The reference always contains three #
characters.)
with:
-
[repository]
: Name of the Git repository. -
[project]
: Path of the directory of the Cucumber project (i.e. the directory containing thepom.xml
file).
This parameter is optional, i.e. if thepom.xml
file is at the root of the repository, this path is absent. -
[feature_file]
: Path and name of the Cucumber test file, from the previous directory (with the.feature
extension). -
[feature_name]
: Feature name as specified in the Cucumber test file.
This parameter is optional, i.e. the string[feature_name]
may be empty. -
[scenario]
: Scenario name as specified in the Cucumber test file.
This parameter is optional, i.e. the string[scenario]
may be empty.
Example: repo/path/to/cucumber/project#path/to/test.feature#feature_name#scenario_name
Old syntax
Old versions of Squash Orchestrator used the following format:
[repository]/[feature_file]#[feature_name]#[scenario]
or
[repository]/[feature_file]#[feature_name]
or
[repository]/[feature_file]
with:
[repository]
: Name of the Git repository.[feature_file]
: Path and name of the Cucumber test file, from the root of the project (with the.feature
extension).[feature_name]
: feature name as specified in the Cucumber test file.
This parameter is optional, i.e. the last two components of the test reference may be absent.[scenario]
: scenario name as specified in the Cucumber test file.
This parameter is optional, i.e. it may be absent.
Example: my_repo/path/to/my/test.feature#my_feature_name#my_scenario_name
This syntax is deprecated and should not be used. It is nevertheless still supported.
This syntax did not support Cucumber tests which are not at the root of the code repository.
Automated test reference and execution
The feature [feature_name]
and scenario [scenario]
precisions in the test reference have no impact on the execution, but only on the test result determination.
So, even when defining the specific level of the feature or scenario, all tests defined in the .feature file will be executed (this means, for example, that if several Squash TM test cases point toward the same file but with different features or scenarios, then all the file tests will be executed several times).
The test reports include the traces of all executed tests. But, only the Error / Failed / Success statuses corresponding to the specific feature or scenario are used to determinate the test case result.
Determination of the result of the test case
If a feature name [feature_name]
and/or a scenario name [scenario]
is not specified, the result of the Squash TM test case is calculated by taking into account the individual results of each executed scenario:
- If at least one scenario has an Error status (in case of a technical issue), the status of the execution will be Blocked.
- If at least one scenario fails functionally and none of the other ones has an Error status, the status of the execution will be Failed.
- If all scenarios succeed, the status of the execution will be Success.
No support of cucumber-junit-platform-engine
The current implementation of Squash Orchestrator uses the mvn -Dcucumber.features=path/to/mytest.feature
option to specify the .feature file to be executed.
This option is not managed by cucumber-junit-platform-engine. This results in the fact that launching a Cucumber test, in the orchestrator, executes all the tests of the Maven project.
In order to avoid this issue, the junit-vintage-engine engine must be used.
If your tests are using cucumber-junit-platform-engine, you can modify them to instead use junit-vintage-engine as described here:
- in the
pom.xml
fileshould be replaced by<dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-junit-platform-engine</artifactId> <version>7.2.3</version> <scope>test</scope> </dependency>
<dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-junit</artifactId> <version>7.2.3</version> <scope>test</scope> </dependency> <dependency> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> <version>5.8.2</version> <scope>test</scope> </dependency>
- in the test root file
should be replaced by
import org.junit.platform.suite.api.IncludeEngines; import org.junit.platform.suite.api.SelectClasspathResource; import org.junit.platform.suite.api.Suite; @Suite @IncludeEngines("cucumber") @SelectClasspathResource("path/to/my-feature-files") public class RunCucumberTest { }
import org.junit.runner.RunWith; import io.cucumber.junit.Cucumber; @RunWith(Cucumber.class) public class RunCucumberTest { }
Nature of the exploitable Squash TM parameters
The exploitable Squash TM parameters will differ depending on whether you're using the Community/Premium or Ultimate version of Squash.
Here is a table showing the exploitable parameters (these parameters are transmitted as test parameters, see below, Squash TM does not generate global parameters):
Nature | Key | Community/Premium | Ultimate |
---|---|---|---|
Name of the dataset | DSNAME | ✅ | ✅ |
Dataset parameter | DS_[name] | ✅ | ✅ |
Execution ID | TC_EXECUTION_ID | ❌ | ✅ |
Test case reference | TC_REFERENCE | ✅ | ✅ |
Test case internal UUID | TC_UUID | ✅ | ✅ |
Test case custom field | TC_CUF_[code] | ✅ | ✅ |
Iteration custom field | IT_CUF_[code] | ❌ | ✅ |
Campaign custom field | CPG_CUF_[code] | ❌ | ✅ |
Test suite custom field | TS_CUF_[code] | ❌ | ✅ |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
Availability of the execution ID
TC_EXECUTION_ID
is only available with Squash TM 8.0 or later.
As indicated, Squash TM adds a prefix to the code of the transmitted custom field. Make sure to take it into account.
Refer to the documentation for more information about custom fields.
Parameters usage
It is possible, when running Cucumber tests, to exploit parameters within it. A parameter can be a test parameter or a global parameter. Squash TM transmits only test parameters. Test parameters and global parameters can be used in the case of a launch from a CI/CD pipeline with the cucumber/params
and cucumber5/params
actions.
To do this, the following steps must be followed:
-
Import opentestfactory-java-param-library into the Maven project containing the tests to run by adding to the
pom.xml
file:- the following Maven repository:
<repositories> <repository> <id>org.squashtest.repo.release</id> <name>Squashtest repository for releases</name> <url>https://nexus.squashtest.org/nexus/repository/maven-squashtest-public-releases/</url> </repository> </repositories>
- the following Maven dependency:
<dependencies> <dependency> <groupId>org.opentestfactory.util</groupId> <artifactId>opentestfactory-java-param-library</artifactId> <version>1.1.0</version> </dependency> </dependencies>
-
You can then retrieve the value of a parameter of the desired type using the following syntax:
ParameterService.INSTANCE.getString("paramName"); ParameterService.INSTANCE.getInt("paramName"); ParameterService.INSTANCE.getFloat("paramName"); ParameterService.INSTANCE.getDouble("paramName"); ParameterService.INSTANCE.getBoolean("paramName");
The above methods look for the desired parameter in the test parameters; if they cannot find it, they then look for it in the global parameters.
These methods throw aParameterNotFoundException
if the parameter is not found. If the parameter is found but cannot be converted to the desired type, aParameterFormatException
is thrown. Consider handling these exceptions in your test classes.
Warning: the conversion of Boolean data returnstrue
when the character string to be converted is equal to"true"
(whatever the case),false
in all other cases; but never propagates an exception. -
It is also possible to define a default value in the case where the parameter does not exist by using the following syntax:
ParameterService.INSTANCE.getString("paramName", defaultValue); ParameterService.INSTANCE.getInt("paramName", defaultValue); ParameterService.INSTANCE.getFloat("paramName", defaultValue); ParameterService.INSTANCE.getDouble("paramName", defaultValue); ParameterService.INSTANCE.getBoolean("paramName", defaultValue);
The above methods therefore do not propagate a
ParameterNotFoundException
when the parameter sought is not found but propagate aParameterFormatException
if the parameter is found, but cannot be converted to the desired type. -
It is also possible to target a test parameter or a global parameter with specific methods. As with the previous methods, they are available in versions with and without default values. Here are a few examples:
ParameterService.INSTANCE.getTestString("paramName"); ParameterService.INSTANCE.getGlobalInt("paramName"); ParameterService.INSTANCE.getTestFloat("paramName", defaultValue); ParameterService.INSTANCE.getGlobalBoolean("paramName", defaultValue);
Using the name of a Squash TM dataset as a tag
In addition of the parameter usage described above, Squash is able to use the name of a Squash TM dataset as a tag value, allowing to execute a specific dataset of a Cucumber scenario.
In order to achieve this, you will have to follow these steps:
-
Fill the datasets in the Parameters tab of the test case in Squash TM.
-
Create in a Cucumber scenario as many example table as there are dataset in Squash TM test case. Annotate them with a tag corresponding to the name of a Squash TM dataset.
-
Create one line of elements in each example table to set scenario's parameters values for the dataset.
Example
Below is an example of a Cucumber test file and the corresponding Squash TM test case automation:
Adding parameters to the test launch command line
You can pass additional parameters to the mvn test
command using the
CUCUMBER_EXTRA_OPTIONS
environment variable. Here
is an example of how to define an environment variable in Squash TM and associate
it with the orchestrator.
Some parameters are already defined in the mvn test
command used to
launch tests:
Note
Use the following syntax to pass additional parameters: -Dcucumber.[module].[param]=[value]
(for example, -Dcucumber.ansi-colors.disabled=true
).
mvn test \
-f "{POM_path}" -Dmaven.test.failure.ignore=true \
-Dcucumber.features="{feature_path}" \
-Dcucumber.plugin="html:{html_report_path},junit:{junit_report_path}" \
-Dcucumber.filter.tags="{Squash_TM_tags}" $CUCUMBER_EXTRA_OPTIONS
Note
The CUCUMBER_EXTRA_OPTIONS
environment variable can only be used to define a Cucumber JVM option, not a Maven option.
mvn test \
-f "{POM_path}" -Dmaven.test.failure.ignore=true \
-Dcucumber.options="{feature_path} --plugin html:{html_report_path} \
--plugin junit:{junit_report_path} --tags {Squash_TM_tags} $CUCUMBER_EXTRA_OPTIONS"
You must avoid passing, via the CUCUMBER_EXTRA_OPTIONS
variable, the command line parameters that conflict with the parameters already used, or the parameters that impact the generation or alter the path of the reports expected by the orchestrator (view the report list for Cucumber JVM 4 and for Cucumber JVM 5 or greater).
Non-support of the space character on Linux
Squash Orchestrator currently does not support the space character () in the
CUCUMBER_EXTRA_OPTIONS
environment variable for tests executed in a Linux execution environment.
Generation of Allure reports with the cucumber/cucumber and cucumber5/cucumber actions
When using the cucumber/cucumber
and cucumber5/cucumber
actions in a workflow, if you want the Cucumber test results to appear in the Allure report generated for the workflow, you need to use the JUnit reporter (possibly with other reporters).
Performance boost
You can use Maven Daemon, a Maven wrapper, to speed up the Cucumber test executions: see the technical details chapter. It is useful when there are many short tests to execute. If each test takes over several minutes to run, Maven Daemon is of no real interest.
Supported versions
Squash has been validated with the following versions of Cucumber JVM. More recent versions should work properly.
Testing Technology | Version |
---|---|
Cucumber JVM | 7.0.0 |
Cucumber JVM | 4.2.6 |