Analyze the results

Execution Status of Executions and Automated Suites

Squash Status	Color	Execution	Automated Suite
Ready		The ITPI is part of an automated suite that has been launched, but the corresponding automated script has not yet started.	-
Running		-	The automated suite has been launched.
Passed		The ITPI has been executed and its result (as reported by the test framework) is 'passed'.	All ITPIs in the automated suite have been executed and have a status 'passed' or 'skipped'.
Failed		The ITPI has been executed and its result (as reported by the test framework) is 'failed'.	All ITPIs in the automated suite have been executed, none have a status 'blocked', and at least one ITPI has a status 'failed'.
Blocked		The ITPI could not be executed or its results could not be analyzed: the test does not exist, a library/dependency/… is missing, the Surefire report was not generated…	All ITPIs in the automated suite have been executed, at least one ITPI has a status 'blocked'.
Skipped *		The ITPI was not executed or has a status 'skipped' (i.e., it was marked in the automated test code as not to be executed).	All ITPIs in the automated suite have been executed and have a status 'skipped'.
Cancelled		The ITPI will never be launched because the orchestrator's workflow was deliberately interrupted by a user in the meantime.	The automated suite was cancelled before all the ITPIs in the automated suite were executed.

* The 'Skipped' status currently exists only for Cucumber and Robot Framework.

Squash Orchestrator automated suites

From the Automated Suites anchor of an iteration or a test suite, you can view the list of automated suites.

For each automated suite, the following informations are available:

• creation date;
• status of the automated suite ('Passed', 'Failed', 'Cancelled', 'Blocked', 'Running', 'Skipped');
• details containing the list of suite executions as well as execution reports;
• overall report and execution log;
• user who launched the automated tests;
• type of entity from which the suite was launched (Iteration or Suite);
• modification date;
• total number of executions;
• number of executions with a 'Passed' status;
• number of executions with a 'Failed' status;
• number of executions with another status ('Blocked', 'Cancelled', 'Ready', or 'Skipped');
• tags of execution environments used to launch the tests, by default the technology of the automated test;
• environment variables used to launch the tests.

Cleaning Automated Suites

An administrator, project manager, or referent tester can:

• delete one or more automated suites by clicking the button;
• prune partially (only attachments associated with executions with a 'Passed' status will be deleted) by clicking the button, then selecting "Prune partially";
• prune fully (attachments from all executions will be deleted) one or more automated suites by clicking the button, then selecting "Prune fully";
• stop an automated suite launched from Squash TM with a 'Running' status by clicking the button.

An advanced tester can stop the automated suites they have launched, but not those launched by other users.

Info

Deletion and cleaning of attachments only apply to completed automated suites. Those with a 'Running' status will be ignored.

Consult an automated execution and its execution reports

From the execution plan

You can view a run and its reports from the execution plan.

The run consultation page is displayed by clicking on the Run History submenu and then on the run number.

From the execution consultation page, the tags and the environment variables used are visible and the execution reports can be downloaded in the attachment block.

From an automated suite

It is possible to view the details of an execution and its reports from the automated suite.

You can view the execution reports by clicking on the button. They are organized by test case to make them easier to read.

You can view the execution logs for the automated suite by clicking on the button.

Viewing the Execution Statuses of BDD Test Steps

Info

This feature requires Squash Orchestrator version 4.11.0 (delivery 2024-05) or later to function.

You can view the execution statuses of the test steps from a BDD test execution (Robot Framework or Cucumber) directly from the Execution Scenario anchor of an execution.

Warning

The execution status of test steps will not be updated and will remain "Ready" if the number of test steps in Squash differs from the number of steps in the XML execution report (e.g., if some test steps are missing in the XML report).

The equivalences between the test step statuses and the Squash statuses are as follows:

Squash Status	Cucumber	Robot Framework
Passed	passed	PASS
Failed	failed	FAIL
Skipped	skipped	SKIP
Not run	-	NOT RUN

Comparing Execution Statuses

Warning

Comparing execution statuses is a feature available with the Squash Premium license and the Squash TM Premium plugin.

It is possible to compare the detailed execution statuses of ITPIs from multiple automated suites.

To compare, select at least two automated suites, then click on the icon .

The automated suites comparison pop-up contains a table with one row per ITPI and one column per automated suite.
The suites are sorted by selection order, and the columns can be filtered and sorted by execution status.