TL;DR

When you have fuzzy requirements, it means that requirements are driven by statistics. How to create statistical testing oracles? The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

Informally, a statistical model can be thought of as a statistical assumption (or set of statistical assumptions). That assumption allows us to calculate the probability of any event [source].

For example, in six-sided dice, a statistical assumption is that the probability of each face coming up is 1/6. Or if you are a cheater and you put extra weight into the dice on one side, the possibility of the opposite side coming up would be 1.

In the dice example, your test would be to check a number of side occurrences in a thousand dice throws. In the first case occurrence of number six is approximately 1/6 x 1000, and in the second case, the presence of six is 1000 (with the assumption that extra weight is on number one side).

Advantages

Statistical oracle model allows checking:

• huge data sets
• live systems data
• after the fact

Disadvantages

Using statistical model oracle, we could have false alarm bugs and missed bugs. The reason is the nature of the statistical oracle. For example, the oracle that customers order Big Mac in 90% of every order. This could lead to a missed bug if the system is missing in the catalog subsystem other popular menu items. So those orders are not recorded.

We could have a false positive bug in the case when we have a statistical oracle that 80% of orders are from Europe but we test when is night in Europe.