TL;DR

Here we explain why it is not possible to test every possible input to every input, output, and intermediate variable. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

To test everything, we need to check every possible input to every program variable. We can divide variables into three groups:

1. The input variable is a variable that takes data from a user
2. Output variable – is a variable that is set in the program or by a user. That variable presents to a user it’s content.
3. Intermediate variable.  On the way from input to output variable. Stores intermediate results.

As experienced testers, we would do value sampling. Our test would consist of the smallest and largest values for a variable. We would also include boundary values, nearest invalid, and valid values to the smallest and largest values. It is also possible that values are form smaller groups (for example, user age groups), so we would repeat such sampling for each of those groups.

But with sampling, we consciously do not test with ALL POSSIBLE values. We could miss some bugs. Do you want to miss a bug in airplane navigation system?

How to calculate all valid values? We need to know the memory size of the variable that we use. For example, 64 bits unassigned integer have values in the range [0, 18,446,744,073,709,551,615].

MASPAR Example

Doug Hoffman, in his article Exhausting your test options explains test strategy for the MASPAR processor. He tested one of the implemented mathematical functions, integer square root. The input was 32-bit word unsigned integer with possible values [0, 4,294,967,296].

How many of those input values should we test?

Doug tested all 2^32 possible values, at it took him 6 minutes to compare results with square root oracle. They found two errors, not on boundaries.

For 64 bit unsigned integer, this test is impossible:

2^32 x 6 minutes is 24 billion minutes.

We have more valid inputs:

• edited input – when we enter data for the first time, this could mean that the input variable is created and is in the initial state. But what would happen if we edit input value and start calculation again? And then we repeat that action?
• Variations on input timing – enter data quickly or slowly. Or enter data just before timeout expiration. What would happen if you enter data before, during, or after some other event (operating system auto-update)?

What about invalid inputs:

• we first try boundary values. If input accepts 1 to 100, we try with 0, 2, 99, 101, -1
• if inputs are variables that are multiplicated, we try to create overflow in output or intermediate variable
• we can try overflow with a number of characters that output could display
• empty input could cause the variable to not be initialized or initialized with a NULL value, which usually causes an unhandled exception.
• What about Easter Eggs? While the term Easter egg has been used to mean a hidden object for some time, in reference to an Easter egg hunt, it has come to be more commonly used to mean a message, image, or feature hidden in a video game, movie, or other, usually electronic, medium [Wikipedia].

No user would do that really means No user that I can think of, who I like, would do that on purpose.