Manage Exploratory Testing

Overview

Exploratory testing is a complementary practice to traditional testing. Exploratory testing is not chaotic, random, ad hoc, or simply monkey testing (although the latter is part of the toolkit of exploratory testing). They are structured tests based on observation and experience rather than test scenarios, contributing to product improvement.

A frequently used method to organize exploratory tests is through sessions. These sessions are based on charters that specify the objectives and scope of the tests.

An organizer is responsible for creating, tracking, and reviewing sessions. Participants in the session are responsible for execution.

In Squash, charters for exploratory tests are managed in the Test Cases workspace, while sessions are conducted and reviewed in the Campaigns workspace.

Before the Session

Define an Exploratory Test Charter

The test charter is defined in test cases with the "Exploratory" format by the session organizer or the test manager.

Exploratory test cases have the same attributes as other test case formats: 'Status,' 'Weight,' 'Nature,' 'Type,' 'Format,' 'Description,' as well as custom fields.

They can also be associated with requirements.

However, they cannot be parameterized through parameters and datasets, nor automated.

The main characteristic of exploratory test cases is the test charter. It is usually defined by the test manger, but another actor can also handle it.

It includes two elements:

• Charter
• Duration: it can also be defined when planning a session in the Campaigns workspace

The content of the charter may vary from one team to another but generally includes the following elements:

• Objective: a sentence summarizing the mission of the participants in the session
• Scope:
  • What features and/or non-functional requirements need to be tested?
  • Are there specific points that need to be tested? If so, define them in the form of a checklist.
  • Are there any points outside the scope of the session that should not be tested?
• Test tactics: how should the tests be divided between the participants? Each can play a different "persona." Each can take a different usage workflow and study its variations. Some teams use the six hats method.
• Test environment: which version of the System Under Test (SUT) is deployed on which instance? Which browsers to use? On which operating systems?

It is possible and desirable to add any information that can help with test design during the session: pointers to information sources (user documentation, specifications or user stories, ISO standards or RFCs), anomalies found in the past affecting the session's scope, feedback from developers indicating that certain features are more risky...

A charter can also contain the following elements:

• Session date
• Participants
• Assignment of tests to participants (according to the test tactics defined in the charter)

In Squash, these elements are defined when creating an exploratory test session in the Campaigns workspace.

Create and Prepare an Exploratory Test Session

Once the charter is written, an exploratory test session can be created by the organizer or test manager.

To do this, add the exploratory test case to the execution plan of an iteration or test suite. Once the test case is added, the button allows access to the session consultation page.

This page provides an overview of the session and allows both preparation and tracking.

Before starting the session, the organizer must perform the following actions:

1. Set a date and time as well as a session duration (by default, the duration defined in the test case is used but can be modified in the session). Enter this information in the Information section.

2. Add executions and participants to the session from the Executions section. There are two ways to add and assign executions:

3. Add one execution by clicking the button and then assigning a user to the execution from the corresponding cell in the table.

4. Add multiple executions by clicking the button and selecting the users participating in the session in the popup. An execution is created and assigned to each selected user.

5. Divide the tests of the session among participants according to the defined tactic. This action is performed in the "Division" column of the Executions section.

6. Present the session to the participants. The organizer can use this page to present the session to participants by discussing the charter, division, and duration of the session.

During the Session

Execute an Exploratory Test Session

Once the session is presented to the participants, the organizer can launch it by clicking [Start session]. The execution status of the session is then updated to "In Progress."

Each participant accesses the execution page to which they are assigned by clicking the button in the Executions section of a session.

From this page, the participant can:

• Check general information and requirements related to the test case: Information and Related Requirements sections
• Fill in any custom fields associated to the execution: Information section
• Review the test charter defined in the test case as well as the test division defined in the session: Charter section
• Execute the session: Notes section
• View and link issues to the execution: Issues section

Unlike other test case formats whose execution is based on a predefined scenario, the execution of exploratory tests involves adding notes throughout the session in which the participant briefly describes the steps taken and behaviors observed. It is not about writing precise test scenarios but indicating what was tested and observed during the session.

As exploratory tests do not rely on a predefined scenario, they can be endless and are generally limited in time. To address this, a timer keeps track of the elapsed time for the execution. If a duration has been defined for the session, it is displayed next to the timer. If the defined duration is exceeded, the participant is notified and can choose to stop or continue the execution.

To execute an exploratory test:

1. Start the timer by clicking the button; the progress status of the execution changes to "In Progress"
2. In the Notes section, add a note by filling in the following fields and confirm:
   • the type to qualify the note: Comment, Suggestion, Bug, Question, and Positive Point
   • a rich text to enter observations and tests performed and paste screenshots

3. Once the note is added, it is possible to associate attachments and issues with the note.

   Info

   Notes can be rearranged by drag-and-drop, modified, and deleted. It is also possible to add a note under an existing note via the button of a note.

   The participant can pause the execution at any time via the button. The execution is automatically paused if the participant leaves the page or closes the window.

4. When the execution is complete, stop the timer by clicking the button; the progress status of the execution changes to "Done"

Focus

Depending on the state of the timer and the progress status of the execution, certain actions are possible or blocked:

• execution "in progress", timer started: everything is possible
• execution "in progress", timer paused: modifying, deleting, and reorganizing notes is possible, but adding a note is blocked
• execution "done", timer stopped: reorganizing notes and associating issues with notes is possible, other actions are blocked

Monitor an Exploratory Test Session

During the session, the organizer or test manager can monitor its progress from the session consultation page:

• The table in the Executions section provides an overview of the progress and content of each execution:
  • the note types and number of notes present per type in the execution when hovering over the icons
  • the number of issues associated with the execution
  • the progress status of the execution

• The Activity section displays notes from all executions in reverse chronological order. It is possible to filter notes by user and/or type.

• The table in the Known issues section displays issues associated with all executions of the session.

After the Session

At the end of the session, the organizer indicates that it is finished by clicking the [End session] button.
The session review and finalization phase can then begin.

Conduct the Session Review

The session review is generally conducted by the organizer or test manager.

The principle of the review is to read the notes from each execution of the session to verify that the different points covered in the charter have been explored and to identify questions to transmit to those who can answer them.

To conduct the review:

1. Read the notes from the page of each execution or from the Activity section of the session. To facilitate reading, notes can be filtered by type or by user.

2. In the Review section on the page of each execution, indicate that the review has been conducted:

   • check the "Review done" box
   • add comments if necessary

   

   The review status of executions is visible from the table in the Executions section of the session.

3. When the review of each execution is conducted, the session review is considered complete.

Finalize the Session

To finalize the session, the organizer or test manager must perform the following actions:

• In consultation with the participants, add global comments to assess the session in the Comments section of the session.
• Define a global execution status for the session in the "Execution Status" field of the Information section of the session. This status is reflected in the execution plan and used in calculating requirement coverage indicators.