TL;DR
This is a report on Testival #59 meetup, our very first online meetup. The concept of the meetup was to have 45 min talk and several lighting talks after the break.
Technical Setup
The sponsor of this meetup was Microblink. They provided a link to Google Meet with the maximum number of 150 participants. Google Meet did not fail us. It was easy to join the meeting room and share our desktop if needed. The Break To Rooms feature would also is precious for the Open Session Testival conference. We used meet chat for posting Q&A questions. There were 31 participants, and four of them were international.
The Many Flavours Of QA
Ines and Mateja from Microblink gave this talk. Ines is QA and Mateja is the project manager in R&D. Fascinating combination for the talk presenters. Mateja explained what their product is and the development process of adding a new feature.
The product is a Machine Learning system for image processing. Images are an ID card that is scanned and automatically transformed to customer ID data. A Machine Learning system is a model that is trained with data. Having good test data is very important to create a good model with a small number of false positives and negatives. Company innovation is machine learning models.
Machine Learning model classifies and annotates new ID images based on a model trained using other ID images. For example, the model is trained with 10 Croatian ID cards. New ID cards from the USA must not be classified as a Croatian ID card.
The only Developer problem is to train the model using training data. Requirements are annotation rules, and sometimes those are miscommunicated to developers. Each model with a new feature has its own branch.
Test In Development
Yes, developers do tests in Microblink! First, there are code reviews, the are doing a two-tier review. First-tier is junior level, and the second-tier is senior level. Developer tests are organized in the triangle. They start with unit tests, where the unit is class or method. They do a black box and white box approaches. The white box is a classical developer testing effort, but developers’ black-box approach is not so usual. For that, you need to know how to do skillful test design. After that, developers do integration testing. ML model is packed as an SDK with clear interfaces.
The last step is an end to end test on all supported platforms, from mobile to web and browser. QA team fun starts here.
In their context, code coverage is done periodically, with statement, branch, and path base for coverage metrics. Pre-production check is a mem check, automatic hunt for memory leaks (because language is C++), and that takes time.
They also have specific contexts for regression tests. Code changes are not primary worries, but model changes. How model classification and annotation changed on existing test and training data? For example, there could be a boundary value for classifications. Each model has different boundaries. In regression tests, developers need to decide if the new boundary a bug or a feature.
Tools
QA team problem is a great number of possible environments.
Product Team
The ultimate function test is done by the product team. They need to decide if a new model better than the previous one. And they hold the keys, what better means. For that, they use experience and hard rules.
Platforms
Basic categories of SDK platforms are Core, Web, and Mobile. QA team tests on all those various platforms. Jira and Xray extension is what brings developers and testers together.
Q&A
In the Q&A session, we found out some interesting facts. Jenkins runs all tests in two hours, while with memory checks, it takes six hours. OCR is the most complex model. To reduce the noise in training images, set the OCR image area is a beneficial technique.
Lighting Talks
Alexander Todorov introduced FOSDEM’21 Testing and Automation devroom online conference.
I presented how to use oracle and heuristic in the Meetup feature image upload.
The requirement is to have at least 1200 x 675 pixels image. This is the boundary for domain variable event featured photo size in pixels. I first uploaded an image with a smaller resolution, and then you can not crop the full image area. My expectation was that it would be possible. That was the first oracle. I tested feature image upload with a heuristic. Let’s try smaller images, and it would be possible to crop to that image’s full size. Oracle was the Mac Preview program Inspector feature that gave me information about image resolution. My intention was to upload a new image for Testival #59 event, but that was not possible because the event has already started. So I needed to create a draft for a new event.
From the above screenshot, you can see that it is not possible to crop to image full size, when image size is less than 1200 x 675 px
Zeljko Filipin was last, and he presented an excellent book: Deep Work.
