What Testers Need from ML/AI by Paul Gerrard was the first session of Online Test Conf Autumn 2020 first day sessions. Which features Machine Learning And Artificial Intelligence needs to have to help software testers in daily work.
What is a useful definition of ML/AI in the context of testing? What features do testers need that can be supported by ML/AI? What is the future for ML/AI in testing?
What Should AI and ML Do For Me
Software testers usually see Machine Learning ML and Artificial Intelligence AI as very complicated algorithms that work almost as magic. But we often forget that ML and AI are tools. This means we should ask ourselves, What should AI/ML do for me as a tester?
The Risk with ML/AI is the same as for any other tool; vendor promises with intent to persuade us to buy their tool. I remember a tool that promises to find all HTML UI regressions with minimal training effort from the tester side.
What is the main feature of ML and AI tools? They try to replace part of our brain with data.
A New Model For Testing
To create ML and AI tools, we need a new model for testing. In that model, the central part is the software tester. We need to figure out how our brain is working while we are testing. That mechanism needs to be explained and diagramed.
Mechanical Software Testing Tools
What are mechanical software testing tools? Those are tools that help us with logistics, but not with our thinking. For example, those tools do not know how to integrate software. Integrate software, on the lowest level, means to understand how to connect two lines of code. Jenkins does not know to set up application configuration values for production or test by itself. The software tester knows that and creates a script for that purpose. Software testers create realistic scenarios and supporting data for system testing. That’s even out of the domain of software developers. Not that they are not capable of learning scenario system testing.
Challenge is to fit AI and ML to software tester thinking and doing.
Compare AI and ML
What is the difference between AI and ML?
Future Of Software Testers
But what is our future if we create such tools soon?
How We Think As Testers
Can software testing models help us to understand how we think? Agile Testing Quadrant, Testing Pyramid and Testing V-Model are all testing logistic help.
A New Model For Testing
We need to separate logistics from thinking. So Paul proposes a new model for testing:
A New Model has three parts, left, center, and right:
To simplify, we can state that left are requirements and testing those requirements. Center is about software testing models that are blueprints for testing and automation.
I comment on this because exploratory testing (or testing) spans through all three parts, from the left through the center to the right part.
The Best Tools
For example, Visual Studio Code integrates with various automation frameworks. Record keeping tool is TestRail, but I do now know a tool that supports human thinking and collaboration at the same time (Trello supports just collaboration).
Two Dimensions Of Tools Capability
Software testing tools have two important capabilities, capture knowledge and investigation.
Today we have progress in tools that capture knowledge. ML and AI should help us in investigations. Vendors must be aware that the challenge is not test execution:
What is exactly integration in software testing:
We have vertical and horizontal integration:
And finally, Paul gives us a quadrant of how effectively test. Four dimensions are validation (are we building the right system), verification (are we building the system right), business flow (user perspective), and system flows (technical perspective):
Realism in software testing is a green arrow from left bottom to top right.
ML and AI should take the best from both testing styles, exploratory and structured. The testing tool should be able to capture the tester’s models and activity in real-time. While creating a model, software testers are doing system surveying. The general idea is:
A scalable automatable process of Humans that make the early maps and tools that will follow those human trails.
Interactive Test Environment
The tool should be an interactive test environment with the following features:
Quote Of The Day
Let go of late manual checking.
Testivator Session Score
This is my first conference talk, that I used Testivator for making notes. If you would like to see my note flow during the talk, register to Testivator and let me know in the comments your registered email. I will share the report with you in Testivator.
For this session, I was using the Testivator Mobile application. I took 58 notes and 23 screenshots.
Here are note types by duration. Till end is 1.1 % and it is the ratio of duration from last note to session end and session duration.