TL;DR

This post will explain why all testing could not be automated. All testing could not be automated not because of technical constraints, but because our ability to create oracles is constrained. This post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

Software testers often hear that most (or all testing) should be automated. The code change is pushed, all tests passed in CI, and we are ready to push to live production. What dreamland!

Dr. Cem Kaner says:

Automated testing depends on our ability to programmatically detect when the software under test fails. Automated or not, you must still exercise judgment in picking risks to test against and interpreting the results. Automated comparison-based testing is subject to false alarms and misses. [Exploratory Test Automation – kaner.com. http://kaner.com/wp-content/uploads/2010/08/ExploratoryTestAutomationHoffman.pdf].

Does polyline fill color in Google Draw works?

What could be used as an oracle in this problem?

Let’s use the Mac Digital Color Meter. On the above screenshot, you can see that measured color, in-display native values are:

R:234, G:50, B:35

Standard RGB values for red are:

R: 255, G: 0, B: 0

That is quite a difference. Let’s change display native values:

Switching to Display in Generic RGB, we got:

R: 251, G: 0, B: 6

This is much closer to what red should be.

A difference

Does that difference make a difference for Google Draw? If you are not color blind, you will see polyline fill color from the first screenshot as red. Yes, my dear tester, you need to make a JUDGMENT! Testing is a cognitive activity, not a mechanical one. We can agree that for Google Draw, it does not require that red does meet the R: 255, G: 0, B: 0 standards. By focusing our testing on things that matter to product clients, we simplify testing, making it possible.

Instead of thinking pass vs. fail, consider thinking problem vs. no problem. [Michael Kelly, Using Heuristic Test Oracles.]