Retry💎

Replay failed tests at the end of an execution

Info

This plugin is available only in the Ultimate version of SquashTM.

Starting from version 5.1.0, the Docker image of SquashTM Orchestrator includes the retryer plugin, which allows replaying failed test cases at the end of an execution. This can be useful when an automated test suite contains flaky tests.

To use this feature, simply set up a hook:

hooks:
- name: Retry Hook
  events:
  - channel: teardown
  before:
  - uses: tm.squashtest.org/retry@v1
    with:
      max-retry: 2
      scope: test.technology == 'postman'
      failure-status: ['FAIL']

Warning

retry action can only be used at the teardown channel hook level, which is executed at the end of each workflow job.

This example hook will replay all Postman tests with a FAIL status twice at the end of each workflow job. If all failed tests pass after the first replay, the second replay is not triggered.

This hook calls the tm.squashtest.org/retry@v1 action of the retryer plugin.
This action has three optional parameters:

• max-retry: the number of retries for failed tests (defaults to 1).
• scope: the scope applied to retried tests (defaults to 'true', meaning all executed tests are considered for replay).
• failure-status: the statuses of tests considered as failed (defaults to ['FAIL', 'ERROR']).

failure-status possible values are FAIL, ERROR, SKIPPED, and PASS.

The scope uses functions and expressions described in the OpenTestFactory documentation. It relies on the orchestrator's test context, whose properties are detailed in the Quality Gate documentation.

retry actions can be chained. In this case, each action takes into account the results of previous replays. Let's assume the following chain of retry actions is applied to a workflow with a single job containing Postman and Cypress tests:

hooks:
- name: Chain Retry Hooks
  events:
  - channel: teardown
  before:
  - uses: tm.squashtest.org/retry@v1
    with:
      failure-status: ['FAIL']
  - uses: tm.squashtest.org/retry@v1
    with:
      scope: test.technology == 'postman'
      failure-status: ['FAIL', 'ERROR']
  - uses: tm.squashtest.org/retry@v1
    with:
      max-retry: 2
      scope: test.technology == 'postman' && test.importance == 'HIGH'
      failure-status: ['FAIL', 'ERROR']

At the end of the workflow, the first action replays all tests with a FAIL status. The second action retries Postman tests that remained in FAIL after the first replay, as well as all Postman tests in ERROR that were not within the scope of the first action. Finally, the third action will attempt to retry twice all high-importance Postman tests whose status has not changed after the second replay.