Skip to content

Run automated tests in CI/CD

Now that your SquashTM environment is ready, configure your CI/CD pipeline to run your automated test suite and generate the necessary artifacts.

What you'll learn in this section:

  • Run your automated tests in the CI/CD pipeline
  • Generate test reports in a parsable format (e.g., Robot Framework output.xml, JUnit XML, Cucumber JSON, etc.)
  • Save artifacts for the next steps (parsing and publishing to SquashTM)

Example: GitLab CI/CD + Robot Framework

In this example, we'll configure a GitLab CI/CD pipeline to automatically run Robot Framework tests and generate the test reports needed for SquashTM.

In the previous section (Prepare SquashTM), you created a project, test cases with automated test references, and an execution plan (iteration) that will collect your automated test results. Now that your SquashTM environment is ready, let's wire a simple pipeline that runs tests and produces an xUnit report.

See the Complete File

You can view the complete (final) .gitlab-ci.yml file here: .gitlab-ci.yml.

Step 1: Create the pipeline configuration file

GitLab CI/CD pipelines are defined in a file called .gitlab-ci.yml located at the root of your repository.

Create this file in your project's root directory:

your-project/
├── .gitlab-ci.yml    ← Create this file
├── tests/
│   └── login_validation_tests.robot
└── ...

Step 2: Define the pipeline structure

Open .gitlab-ci.yml and start by defining the basic pipeline structure:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - when: never

variables:
  ROBOT_VERSION: "7.4.1"

default:
  image: python:3.14-slim

stages:
  - test

What this does:

  • workflow.rules: Controls when the pipeline runs:
    • Runs automatically on merge requests
    • Runs on direct pushes to branches (but avoids duplicates with merge requests)
    • Allows manual runs from the GitLab UI
    • Supports scheduled pipelines
    • Blocks all other pipeline sources for security
    • For quick testing, you can simplify this to: 
      workflow:
  rules:
    - when: always
  • variables: Defines ROBOT_VERSION used to pin the Robot Framework version
  • default.image: Uses a lightweight Python 3.14 Docker image to run the pipeline
  • stages: Defines one stage (for now): test (run tests)

Step 3: Add the test execution job

Now add a job to run your Robot Framework tests:

robotframework-tests:
  stage: test
  allow_failure: true
  script:
    - python --version
    - pip install --upgrade pip --root-user-action=ignore
    - pip install robotframework==${ROBOT_VERSION} --root-user-action=ignore
    - robot --xunit xunit.xml ./tests/
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - output.xml
      - log.html
      - report.html
      - xunit.xml
    reports:
      junit: xunit.xml

What this does:

  • stage: test: This job runs in the test stage
  • script: Commands to execute:
    • Check Python version
    • Install/upgrade pip
    • Install Robot Framework with the version defined by ROBOT_VERSION
    • Run Robot Framework tests and generate a JUnit report xunit.xml (the example uses xunit.xml for parsing)
  • artifacts: Save test reports for later use:
    • when: always: Save artifacts even if tests fail
    • expire_in: 7 days: Keep artifacts for 7 days
    • paths: List of files to save
    • reports.junit: Display test results in GitLab's UI

Understanding the Robot Framework command

The key command that runs your tests is:

robot --xunit xunit.xml ./tests/

Command breakdown:

  • robot: The Robot Framework test runner
  • --xunit xunit.xml: Generate a JUnit-style XML report (for GitLab UI and parsing)
  • ./tests/: Run all tests in the tests directory

Complete pipeline file

At this stage, you have a pipeline that can run tests and save artifacts in your GitLab repository. 

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - when: never

variables:
  ROBOT_VERSION: "7.4.1"

default:
  image: python:3.14-slim

stages:
  - test

robotframework-tests:
  stage: test
  allow_failure: true
  script:
    - python --version
    - pip install --upgrade pip --root-user-action=ignore
    - pip install robotframework==${ROBOT_VERSION} --root-user-action=ignore
    - robot --xunit xunit.xml ./tests/
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - output.xml
      - log.html
      - report.html
      - xunit.xml
    reports:
      junit: xunit.xml

Step 4: Commit and verify the pipeline

Save the .gitlab-ci.yml file and commit it to your repository:

git add .gitlab-ci.yml
git commit -m "Add CI/CD pipeline for automated tests"
git push

Verify the pipeline

  1. Go to your GitLab project
  2. Navigate to Build → Pipelines
  3. You should see a new pipeline running automatically (if not, you can run it manually by clicking the Run pipeline button)
  4. Click on the pipeline to view the robotframework-tests job
  5. Click on the job to see the tests running and view the logs
  6. Once complete, you can download the artifacts (including output.xml and xunit.xml)

What you've accomplished

Your CI/CD pipeline now:

  • Automatically runs on every commit
  • Executes all Robot Framework tests
  • Generates automated test reports
  • Saves artifacts for 7 days

Next step

Now that you have test reports being generated automatically, the next step is to:

  1. Parse the test report - Convert a report file into a JSON format that SquashTM can understand
  2. Publish to SquashTM - Send the parsed results to your SquashTM instance via the API

Proceed to: Parse the report