Today I visited the SAST Autumn 23 Conference, gathering great presentations on the topic “Test and AI”. I also asked ChatGPT for the first time….
AI, Large Language Models, Generative Pre-Trained Transformers
AI has been around for many years, dating back to the first part of the last century when the first attempts to make machines perform tasks that would typically require human intelligence were made.
Now, if you ask Google and ChatGPT the question “When was AI introduced for the first time”, you get slightly different answers….this because the two use different techniques and models; Google choses a suitable language model and transformers, while ChatGPT uses one large Generative Pre-Trained Transformer.
One thing to remember is, that asking a question the old-fashioned way using e.g. Google or Bing instead of e.g. ChatGPT, has a much smaller environmental impact. The deep-learning of a Generative Pre-Trained Transformer consumes much higher computational resources than simpler Large Language Models.
AI in Testing?
Well, AI is not really that mature yet and the support in test tools is not high. Especially when it comes to cognitive testing, the available tools have just emerged and offers very basic capabilities at the moment. The trickiness lies within the modeling capabilities and that there is often too little computable data available. This causes the deep-learnings to produce skew models.
The reason for this is that the domain knowledge needed to train the models is still in the human brains – the superusers.
When looking at Gartner’s “Hype cycle“, it becomes obvious that generative AI is still a hype.
Mariya Tuleika presented a “AI test pyramid”, with the traditional test tools/frameworks Selenium, Playwright and Cypress at the base, followed by the AI-based test solutions Testim, OpenAI, TestRiger, Applitools, TestGraft and the cognitive testing at the top with the freshlings Katalon Studio and Microsoft Cognitive Services.
A reflection from me and my colleagues focusing on continuous testing- and deploy, is that very few are supported by ALM tools, especially Azure DevOps. At least if one would like to do configuration-as-code, not to speak about on-prem infrastructure-as-code.