Manage artificial intelligence servers
Warning
The generation of test cases by artificial intelligence is available with Squash Ultimate license and Squash TM Premium plugin.
It is an experimental feature: it works, but it may not be powerful enough to be used in production.
The aim is to allow users to experiment with generating test cases using artificial intelligence, in order to determine whether this type of functionality might be of interest to them and, if so, to provide Henix with any feedback regarding the integration of AI into Squash.
Add, delete an artificial intelligence server
From the Artificial intelligence servers table, accessible by clicking on the icon, you can add
or delete
one or several servers.
Deleting a server linked to a project removes the link between the project and the AI server.
When creating a server, the Name and URL fields must be filled in. The URL must correspond to the URL of the artificial intelligence server API. This information can be found in the API documentation of the chosen server.
For example, for Mistral, the URL is https://api.mistral.ai/v1/chat/completions
.
By clicking on the ID # or the Name of a server, its consultation page is displayed, allowing its full configuration.
Several actions can be done from the AI server consultation page:
- modify the server Name, URL and Description;
- set the server authetication Token;
- configure the server by setting the Payload template and the JSON path for the server response;
- fill in a Sample requirement in order to test the server's configuration;
- delete the AI server using the [...] button.
Configure an artificial intelligence server
It is essential to configure your server so that you can use it in projects to generate test cases based on a requirement.
Add an identification token to the server
For artificial intelligence servers, authentication is performed using a token. For some servers, this token is not necessary: in this case, leave the field empty.
Configure the server
In order to contact the server, Squash needs to know the template of the request body expected by the AI server. To find out which parameters to enter, consult the documentation of that server.
Some data is absolutely essential for test case generation to work:
-
The JSON format of the expected return:
(Which must be encoded as the string{ "testCases": [ { "name": "", "description": "", "prerequisites": "", "testSteps": [ { "index": 0, "action": "", "expectedResult": "" } ] } ] }
\"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\"
.)
Squash can only interpret JSON responses which respect this schema. -
The
{{requirement}}
placeholder: it allows the request body to be populated later with the selected requirement to create test cases.
Test the server configuration
To test that the configuration is working correctly, you need to provide an example of requirement.
Click on Test the configuration: a request is sent to the chosen server. If the configuration is correct, a pop-up displays the test cases generated.
If an error occurs, it is recorded under the requirement, so that the problem can be analysed and resolved.
If necessary, adjust the Top-P
, Top-K
, Temperature
… parameters (depending on the AI model chosen) to obtain better results.
Configuration examples
OpenAI
gpt-4-1106-preview
is one of the many models offered by OpenAI.
URL: https://api.openai.com/v1/chat/completions
Payload template:
{
"model": "gpt-4-1106-preview",
"messages": [
{
"role": "system",
"content": "You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\". Your answer must contain only that JSON! The first step of a test must be numbered 0."
},
{
"role": "user",
"content": "Write test cases for the following requirement: {{requirement}}"
}
],
"temperature": 0.9,
"top_p": 0.9,
"n": 1,
"stream": false,
"max_tokens": 2000,
"presence_penalty": 0,
"frequency_penalty": 0
}
JSON path of generated text: choices[0].message.content
Mistral
mistral-large-latest
is a model offered by Mistral.
URL: https://api.mistral.ai/v1/chat/completions
Payload template:
{
"model": "mistral-large-latest",
"messages": [
{
"role": "system",
"content": "You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: \"{\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}\". Your answer must contain only that JSON! The first step of a test must be numbered 0."
},
{
"role": "user",
"content": "Write test cases for the following requirement: {{requirement}}"
}
],
"response_format": {
"type": "json_object"
},
"temperature": 0.7,
"max_tokens": 2048,
"stream": false,
"safe_prompt": false,
"random_seed": null
}
JSON path of generated text: choices[0].message.content
Mistral-7B-Instruct at Hugging Face
mistralai/Mistral-7B-Instruct-v0.3
is a small "open-weight" model created by Mistral and freely usable at Hugging Face.
It is probably not suitable for complicated requirements, but it is included here for those who want to experiment with AI generation for free (however, you need to create an account and an access token at Hugging Face). It is also an example of a different configuration from those above.
URL: https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3
Payload template:
{
"inputs": "<s>[INST]You are an expert in manual software testing. You are in charge to define the test cases necessary to validate a specific requirement. You must provide the test cases formatted in JSON as an array with the following format: {\"testCases\":[{\"name\":\"\",\"description\":\"\",\"prerequisites\":\"\",\"testSteps\":[{\"index\":0,\"action\":\"\",\"expectedResult\":\"\"}]}]}. Your answer must contain only that JSON! The first step of a test must be numbered 0. Write test cases for the following requirement: {{requirement}}[/INST]</s>",
"parameters": {
"return_full_text": false,
"temperature": 1.0,
"max_new_tokens": 4096
}
}
JSON path of generated text: [0].generated_text