Squash TM test execution plan retrieval with a workflow
Squash gives you the possibility to insert, in a CI/CD pipeline, a step to retrieve and execute an automated test plan defined in Squash TM using a workflow written in YAML or JSON. In the case of Jenkins, the configuration of this step can be simplified with a plugin, see the corresponding page of this guide.
You can find more information regarding the redaction of a workflow in the OpenTestFactory Orchestrator documentation.
Prerequisites
A user belonging to the Test automation server group must exist in Squash TM.
Integration of the Squash TM execution plan retrieval step into a workflow
The execution plan, iteration or test suite, must contain at least one test plan item, see the Squash TM glossary) linked to an automated test case, as described in the instructions here.
In order to retrieve this execution plan from Squash TM with a workflow, you need to call the corresponding generator action.
Here is a simple example of such a workflow in JSON format:
{
"apiVersion": "opentestfactory.org/v1alpha1",
"kind": "Workflow",
"metadata": {
"name": "Simple Workflow"
},
"defaults": {
"runs-on":"ssh"
},
"jobs": {
"explicitJob": {
"runs-on":"ssh",
"generator":"tm.squashtest.org/tm.generator@v1",
"with": {
"testPlanUuid":"1e2ae123-6b67-44b2-b229-274ea17ad489",
"testPlanType":"Iteration",
"squashTMUrl":"https://squashtm.example.com/squash",
"squashTMAutomatedServerLogin":"tfserver",
"squashTMAutomatedServerPassword":"tfserver",
"technologyLabels":{
"ranorex": ["windows","ranorex"],
"robotframework": ["linux","robotframework"],
"junit": ["linux","junit"]
}
}
}
}
}
Mandatory Fields
A Squash TM generator step must contain the following parameters:
squashTMUrl
: Public URL of Squash TM.testPlanType
: Refers to the type of test plan to retrieve in Squash TM. Only Iteration and TestSuite values are accepted.testPlanUuid
: Refers to the UUID (Universally Unique Identifier), see the dedicated documentation of the desired test plan. This can be found in the Information block of the desired iteration or test suite in Squash TM.
For API token authentication:
squashTMAutomatedServerToken
: API token of the user from the Test Automation Server group to be used in Squash TM;
For more information on creating an API token for a user in the Test Automation Server group, please refer to the dedicated documentation.
For basic authentication (login/password):
squashTMAutomatedServerLogin
: Username of the user from the Test Automation Server group to be used in Squash TM.squashTMAutomatedServerPassword
: Password of the user from the Test Automation Server group to be used in Squash TM.
Optional Fields
tagLabel
: Specific to the Ultimate version π - It refers to the name of the tag type custom field on which the test cases to retrieve are to be filtered.
It is not possible to specify more than one.tagValue
: Specific to the Ultimate version π - It refers to the value of the tag type custom field on which the test cases to retrieve are to be filtered.
It is possible to specify multiple ones separated by|
(examplevalue1|value2
). There has to be at least one value specified for the test case to be taken into account.
Focus
If one of the two tagLabel or tagValue fields is present, the other one must also be specified.
technologyLabels
: To use if you retrieve an execution plan containing tests which must be executed on different environments.
It enables you to specify, for each automation framework, targeted environment execution's tags.
They will be added to the ones specified in the job's runs-on attribute.
For a technology, tags are defined by adding an entry to the field list with format"technology prefix": ["tag1", "tag2"]
.
Info
By default, jobs created by the Generator after retrieving an execution plan are tagged with the corresponding technology prefix (technology name in lower case with no space, for example "robotframework" for Robot Framework) in addition of the ones specified in Generator job's runs-on attribute.
Execution order of tests
The only order assured by Squash is that, for a given Git repository in a given iteration/test suite, the tests will be executed in the order defined in Squash TM.
If a test iteration/suite contains automated tests belonging to multiple Git repositories, the order of execution of tests from one repository relative to the execution of tests from another repository is undefined.
If several test iterations/suites are launched from Squash TM (resp. the CI/CD pipeline), the order of execution of the tests of one iteration/suite relative to the execution of the tests of another iteration/suite is undefined. (Except in the trivial case where a suitable execution environment is available and the first suite/iteration was able to start before the Squash TM user β resp. the CI/CD pipeline β launches another one.)
Usage of the orchestrator with inception
execution environment
OpenTestFactory, and therefore Squash, have a special execution environment called inception
to use an alternative use mode for generator
type jobs.
What is the purpose of inception
environment?
Usual execution process of a generator
job is to retrieve an execution plan from Squash TM and then execute automated tests of this plan on dedicated execution environment.
When an inception
execution environment is used by a generator
job, there is no test execution following execution plan retrieval. Instead, result publication step to Squash TM is run based on a set of reports sent upstream to the orchestrator.
How to use inception
environment in a workflow
Below is a workflow example in YAML
format using inception
execution environment and retrieving a Squash TM execution plan.
metadata:
name: Test Inception
resources:
files:
- report1
- report2
- report3
jobs:
prepare:
runs-on: inception
steps:
- uses: actions/prepare-inception@v1
with:
report.html: ${{ resources.files.report1 }}
output.xml: ${{ resources.files.report2 }}
log.html: ${{ resources.files.report3 }}
robot:
runs-on: inception
needs: [prepare]
generator: tm.squashtest.org/tm.generator@v1
with:
squashTMUrl: https://squashtm.example.com/squash
squashTMAutomatedServerLogin: ${{ variables.SQUASH_USER }}
squashTMAutomatedServerPassword: ${{ variables.SQUASH_PASSWORD }}
testPlanUuid: 1e2ae123-6b67-44b2-b229-274ea17ad489
testPlanType: Iteration
There are several differences compared to the workflow of the previous section:
-
The
files
section in theresources
part: it lists the reports provided to the orchestrator. Information published to Squash TM will be extracted from these reports. -
A preparation job, called
prepare
in this example: it usesactions/prepare-inception@v1
action provider and allows to indicate what reports, usually generated by the test execution, match with the reports indicated in thefiles
section.
The list of reports supported by Squash Orchestrator is indicated on the page corresponding to each test technology:- Agilitestπ
- Cucumber JVM
- Cypress
- JUnit
- Katalonπ
- Playwright
- Postman
- Ranorexπ
- Robot Framework
- SKF
- SoapUI
- UFTπ
The first report in the list (which is the Surefire report for most test technologies) must be present for the orchestrator to determine the test result.
-
The
generator
job: it is very similar to the one of the previous section with two differences:- It uses
inception
environment in theruns-on
field - It specifies that the preparation job is a prerequisite for its execution with the line
needs: [prepare]
- It uses
Such a workflow can be run with the following command:
curl -X POST \
-H "Authorization: Bearer ${TOKEN}" \
-F workflow=@Squashfile \
-F report1=@report1.html \
-F report2=@report2.xml \
-F report3=@report3.xml \
-F variables="SQUASH_USER=${USER}\nSQUASH_PASSWORD=${PASSWD}" \
http://orchestrator.example.com/workflows
curl -X POST ^
-H "Authorization: Bearer %TOKEN%" ^
-F workflow=@Squashfile ^
-F report1=@report1.html ^
-F report2=@report2.xml ^
-F report3=@report3.xml ^
-F variables="SQUASH_USER=%USER%\nSQUASH_PASSWORD=%PASSWD%" ^
http://orchestrator.example.com/workflows
curl -X POST `
-H "Authorization: Bearer $Env:TOKEN" `
-F workflow=@Squashfile `
-F report1=@report1.html `
-F report2=@report2.xml `
-F report3=@report3.xml `
-F variables="SQUASH_USER=$Env:USER\nSQUASH_PASSWORD=$Env:PASSWD" `
http://orchestrator.example.com/workflows
Note in this command the parameters specifying the reports to be sent by the orchestrator to Squash TM.
A workflow using inception
execution environment can also be run from a Jenkins pipeline thanks to the Jenkins plugin step.
Squash TM parameters to exploit in an automated test
By executing a workflow retrieving a Squash TM execution plan, Squash TM passes various pieces of information on execution plan items that can be exploited during test case execution.
The following frameworks can exploit Squash TM parameters during test execution:
- Cucumber JVM
- Agilitestπ
- Cypress
- JUnit
- Katalonπ
- Playwright
- Postman
- Ranorexπ
- Robot Framework
- SKF
- UFTπ
Each of these pages describes the parameters that can be retrieved by the test script and how to retrieve them.
Results publication in Squash TM at the end of the execution
The results and reports of the executed tests are transmitted and accessible in Squash TM in the same manner as for an execution initiated from it, refer to this paragraph.