Skip to content

Manage artificial intelligence servers

Warning

The generation of test cases by artificial intelligence is available with Squash Ultimate license and Squash TM Premium plugin.
It is an experimental feature: it works, but it may not be powerful enough to be used in production.
The aim is to allow users to experiment with generating test cases using artificial intelligence, in order to determine whether this type of functionality might be of interest to them and, if so, to provide Henix with any feedback regarding the integration of AI into Squash.

Add, delete an artificial intelligence server

Manage AI servers' page

From the Artificial intelligence servers table, accessible by clicking on the Icon for artificial intelligence icon, you can add Add or delete Delete one or several servers.

Deleting a server linked to a project removes the link between the project and the AI server.

Add an AI server

When creating a server, the Name and URL fields must be filled in. The URL must correspond to the URL of the artificial intelligence server API. This information can be found in the API documentation of the chosen server.
For example, for Mistral, the URL is https://api.mistral.ai/v1/chat/completions.

By clicking on the ID # or the Name of a server, its consultation page is displayed, allowing its full configuration.

AI server view page

Several actions can be done from the AI server consultation page:

  • modify the server Name, URL and Description;
  • set the server authetication Token;
  • configure the server by setting the Payload template and the JSON path for the server response;
  • fill in a Sample requirement in order to test the server's configuration;
  • delete the AI server using the [...] button.

Configure an artificial intelligence server

It is essential to configure your server so that you can use it in projects to generate test cases based on a requirement.

Add an identification token to the server

AI server authentication block

For artificial intelligence servers, authentication is performed using a token. For some servers, this token is not necessary: in this case, leave the field empty.

Configure the server

ai-server-configuration-block.jpg

In order to contact the server, Squash needs to know the template of the request body expected by the AI server. To find out which parameters to enter, consult the documentation of that server.

Some data is absolutely essential for test case generation to work:

  1. The JSON format of the expected return:

    {
  "testCases": [
    {
      "name": "",
      "description": "",
      "prerequisites": "",
      "testSteps": [
        {
          "index": 0,
          "action": "",
          "expectedResult": ""
        }
      ]
    }
  ]
}
    (Which must be encoded as the string \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\".)
    Squash can only interpret JSON responses which respect this schema.

  2. The {{requirement}} placeholder: it allows the request body to be populated later with the selected requirement to create test cases.

Test the server configuration

AI server test configuration block

To test that the configuration is working correctly, you need to provide an example of requirement.

Click on Test the configuration: a request is sent to the chosen server. If the configuration is correct, a pop-up displays the test cases generated.

AI server generation success pop-up

If an error occurs, it is recorded under the requirement, so that the problem can be analysed and resolved.

ai-server-generation-fail.jpg

If necessary, adjust the Top-P, Top-K, Temperature… parameters (depending on the AI model chosen) to obtain better results.

Configuration examples

OpenAI

gpt-4-1106-preview is one of the many models offered by OpenAI.

URL: https://api.openai.com/v1/chat/completions

Payload template:

{
  "model": "gpt-4-1106-preview",
  "messages": [
    {
      "role": "system",
      "content": "You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\". Your answer must contain only that JSON! The first step of a test must be numbered 0."
    },
    {
      "role": "user",
      "content": "Write test cases for the following requirement: {{requirement}}"
    }
  ],
  "temperature": 0.9,
  "top_p": 0.9,
  "n": 1,
  "stream": false,
  "max_tokens": 2000,
  "presence_penalty": 0,
  "frequency_penalty": 0
}

JSON path of generated text: choices[0].message.content

Mistral

mistral-large-latest is a model offered by Mistral.

URL: https://api.mistral.ai/v1/chat/completions

Payload template: 

{
  "model": "mistral-large-latest",
  "messages": [
    {
      "role": "system",
      "content": "You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\". Your answer must contain only that JSON! The first step of a test must be numbered 0."
    },
    {
      "role": "user",
      "content": "Write test cases for the following requirement: {{requirement}}"
    }
  ],
  "response_format": {
    "type": "json_object"
  },
  "temperature": 0.7,
  "max_tokens": 2048,
  "stream": false,
  "safe_prompt": false,
  "random_seed": null
}

JSON path of generated text: choices[0].message.content

Mistral-7B-Instruct at Hugging Face

mistralai/Mistral-7B-Instruct-v0.3 is a small "open-weight" model created by Mistral and freely usable at Hugging Face.
It is probably not suitable for complicated requirements, but it is included here for those who want to experiment with AI generation for free (however, you need to create an account and an access token at Hugging Face). It is also an example of a different configuration from those above.

URL: https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3

Payload template: 

{
  "inputs": "<s>[INST]You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: {\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}. Your answer must contain only that JSON! The first step of a test must be numbered 0. Write test cases for the following requirement: {{requirement}}[/INST]</s>",
  "parameters": {
    "return_full_text": false,
    "temperature": 1.0,
    "max_new_tokens": 4096
  }
}

JSON path of generated text: [0].generated_text