TL;DR

Software testing without oracles is testing that everybody can do. It is a common misunderstanding that this is software testing. Based on Doug Hoffman’s notes, we will explain how to recognize software testing without oracles, and list its advantages and disadvantages. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

Incompetent Human

There are many ways to be incompetent human in software testing projects. For example, when you start with a new project, you are inadequate regarding the application business domain (e.g., health insurance). Or you are incompetent regarding the technology used on the project (look under the hood of modern web browser).

Your first test strategy is to use the program until it crashes. You do not use any checks, because you do not have a mechanism or principle to recognize the problem with the application under test. Your only oracle is to report that the app crashed.

Ther are advantages to this approach. This is very useful in the early stages of testing. The program had not yet reached the first milestone, but trial without oracles can reveal some significant flows in architecture, third-party components, or program language missing features. Tests are generated randomly or some oversimplified model of application under test.

Disadvantages are that we can notice only spectacular crashes, and miss many essential flaws. It is very hard or impossible to replicate the failure.

Automated tests are usually tests without oracles. They do have checks, but also they are missing a lot of checks.

Competent Humans

Competent humans start with no oracles. They choose tests in such a way that they figure out test correctness by using the output from the test or previous test. They create oracles along the way. Many competent software testers call this exploratory testing, but this is wrong. Skilled testing is running a series of experiments, so every testing is exploratory testing. We state our assumptions based on our heuristics and verify them by running the test.

Advantages are distinct, the application under test will get its oracles based on general or product-specific expectations gathered during the testing.

Disadvantages are that insecure, inexperienced, or dogmatic humans often do not rely on their wits, and they yearn for some type of structure.