TL;DR

What testing makes interesting is that all tests could only provide partial information that might be useful. This sounds like an excuse for inadequate testing. This post will explain why testers that claim the opposite are the ones that are doing inadequate software testing. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

Test oracles can only sometimes provide us with authoritative failures [Hoffman, Kaner].

An example is when we model oracles based on mathematical functions. Mathematical function proof is the authority.

Test oracles cannot tell us whether the program has passed the test, they can only tell us it has not obviously failed [Hoffman, Kaner].

As an example, we can take the function that calculates the square root. Let’s test the square root of number 3. The square root of 3 is an irrational number [wikipedia]. That means it has an infinitive number of digits. If we test square root and output is 1.73, we can ONLY conclude that the program did not obviously fail.

Oracles subject us to two possible classes of errors [Hoffman, Kaner]

• miss – the program failed, but oracle does not expose it

An example is when you check the layout of the web page only on the desktop computer, and you forgot to test it on a smartphone and tablet.

• false alarm – the program did not fail, but the oracle signaled the alarm

In this case, an oracle example could be a statistical model oracle that the number of users is greater than 400, or we have an application problem. This could trigger a false alarm on weekends when application users have better things to do 🙂 although your application is ok.