TL;DR

In the traditional view of software testing, we see testing as a process where we compare the empirical result with the expected result. This post raises four problems with this approach. This post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

When we see software development as a factory assembly line process, it is a common trap to have a view on software testing as:

• Testing is a process of comparing empirical results to expected results
• An oracle is a mechanism for determining whether the program passed or failed a test.

We have at least four problems with that view:

• Our expectations could be wrong.

We expect that the Ctrl + P shortcut on Mac will open the Print dialog. For Mac, this is Command + P.

• Our expectations are not complete.

When I buy something on Amazon my assumption is that I will be charged only for the item price. This could be wrong if my delivery address is in Europe, and I used Amazon.com.

• A mismatch between results and expectations might not be severe enough to report it.

While testing a parking reservation application, we found that the calculated price for the ten years is off for one cent.

• Our expectations are not necessarily credible.

The application will render the project page within 100ms when there are 100 000  projects in the system.

Conclusion

We must be aware that oracles are fallible heuristics. And using those is not lousy testing.