Skip to content

Manage Artificial Intelligence servers

Warning

The generation of test cases by Artificial Intelligence is available with SquashTM Ultimate license and SquashTM Premium plugin.
This is an experimental feature that allows the generation of test cases from a requirement. It will be enhanced in the future, particularly thanks to feedback from our users on this first version.

Add, delete an Artificial Intelligence server

Manage AI servers' page

From the Artificial Intelligence servers table, accessible by clicking on the Icon for Artificial Intelligence icon, you can add Add or delete Delete one or several servers.

Deleting a server linked to a project removes the link between the project and the AI server.

Add an AI server

When creating a server, the Name and URL fields must be filled in. The URL must correspond to the URL of the Artificial Intelligence server API. This information can be found in the API documentation of the chosen server.
For example, for Mistral, the URL is https://api.mistral.ai/v1/chat/completions.

By clicking on the ID # or the Name of a server, its consultation page is displayed, allowing its full configuration.

AI server view page

Several actions can be done from the AI server consultation page:

  • modify the server Name, URL, and Description;
  • set the server authetication Token;
  • configure the server by setting the Payload template and the JSON path for the server response;
  • fill in a Sample requirement in order to test the server's configuration;
  • delete the AI server using the [...] button.

Configure an Artificial Intelligence server

It is essential to configure your server so that you can use it in projects to generate test cases based on a requirement.

Add an identification token to the server

AI server authentication block

For Artificial Intelligence servers, authentication is performed using a token.
For some servers, this token is not necessary: in this case, leave the field empty.

Use of Authorization Header

SquashTM currently only supports authorization via the Authorization: Bearer <token> HTTP header.
This mechanism is very widely supported, but may not be the most advertised by Cloud providers. For example, when using Azure, you need to use the Microsoft Entra ID authentication.

Configure the server

ai-server-configuration-block.jpg

In order to contact the server, SquashTM needs to know the template of the request body expected by the AI server. To find out which parameters to enter, consult the documentation of that server.

Some data is absolutely essential for test case generation to work:

  1. The JSON format of the expected return:

    {
  "testCases": [
    {
      "name": "",
      "description": "",
      "prerequisites": "",
      "testSteps": [
        {
          "index": 0,
          "action": "",
          "expectedResult": ""
        }
      ]
    }
  ]
}
    (Which must be encoded as the string \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\".)
    SquashTM can only interpret JSON responses which respect this schema.

  2. The {{requirement}} placeholder: it allows the request body to be populated later with the selected requirement to create test cases.

Test the server configuration

AI server test configuration block

To test that the configuration is working correctly, you need to provide an example of requirement.

Click on Test the configuration: a request is sent to the chosen server. If the configuration is correct, a pop-up displays the test cases generated.

AI server generation success pop-up

If an error occurs, it is recorded under the requirement, so that the problem can be analysed and resolved.

ai-server-generation-fail.jpg

If necessary, adjust the Top-P, Top-K, Temperature… parameters (depending on the AI model chosen) to obtain better results.

Configuration examples

OpenAI

gpt-4-1106-preview is one of the many models offered by OpenAI.

URL: https://api.openai.com/v1/chat/completions

Payload template:

{
  "model": "gpt-4-1106-preview",
  "messages": [
    {
      "role": "system",
      "content": "You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\". Your answer must contain only that JSON! The first step of a test must be numbered 0."
    },
    {
      "role": "user",
      "content": "Write test cases for the following requirement: {{requirement}}"
    }
  ],
  "temperature": 0.9,
  "top_p": 0.9,
  "n": 1,
  "stream": false,
  "max_tokens": 2000,
  "presence_penalty": 0,
  "frequency_penalty": 0
}

JSON path of generated text: choices[0].message.content

Mistral

mistral-large-latest is a model offered by Mistral.

URL: https://api.mistral.ai/v1/chat/completions

Payload template: 

{
  "model": "mistral-large-latest",
  "messages": [
    {
      "role": "system",
      "content": "You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\". Your answer must contain only that JSON! The first step of a test must be numbered 0."
    },
    {
      "role": "user",
      "content": "Write test cases for the following requirement: {{requirement}}"
    }
  ],
  "response_format": {
    "type": "json_object"
  },
  "temperature": 0.7,
  "max_tokens": 2048,
  "stream": false,
  "safe_prompt": false,
  "random_seed": null
}

JSON path of generated text: choices[0].message.content

Mistral-7B-Instruct at Together AI

mistralai/Mistral-7B-Instruct-v0.3 is a small "open-weight" model created by Mistral and usable at Together AI.
It is probably not suitable for complex requirements, but it is included here for those who want to experiment with AI generation for free (however, you need to create an account, an access token, and get some free credit at Together AI). It is also an example of a different configuration from those above.

URL: https://api.together.xyz/inference

Payload template: 

{
 "model": "mistralai/Mistral-7B-Instruct-v0.3",
 "request_type": "language-model-inference",
  "messages": [
    {"role": "system", "content": "You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\". Your answer must contain only that JSON! The first step of a test must be numbered 0."},
    {"role": "user", "content": "Write test cases for the following requirement:\n{{requirement}}\n"}
  ],
 "temperature": 0.3,
 "max_tokens": 2048
}

JSON path of generated text: output.choices[0].text