TL;DR

This post discusses the fact that 100% code coverage does not mean 100% testing coverage. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

An Example

Consider the following pseudocode:

INPUT A

INPUT B

PRINT SQRT(A) + B

A test with data A=1 and B=1 will cover:

• every statement
• every branch (because there are no visible branches)

But this test will miss a severe bug.

In a study by Brian Marick, 43% of failures were traced to faults of omission (missing code rather than wrong code)[source].

What bug is missed?

To answer that, we need to have source code, documentation, or domain knowledge for sqrt function. With that information, we can write an additional test with values:

A = -1 B = 1

that will find the bug that we do not handle sqrt error code for values < 0. That is missing code. In formal verification for this simple pseudocode, we have a lot of assumptions. The first assumption is that sqrt will return the correct value for all possible A values.

This simple example shows two things:

1. Even with 100% of branch and statement coverage, we could miss essential bugs.
2. It shows the difference between the craft of writing checks and doing testing. Testing is investigating the sqrt function domain through mathematical books or function documentation. We need to know what is statement, multi-conditional, and branch coverage. On the other side, you need to know some programming IDE, checking framework for our pseudo code, and how to compile/run that code.

Many Aspects Of Software

We have to be careful with measuring coverage. There are many aspects of code coverage that could not be measured with statement, branch, multi-conditional, and loop coverage:

• Unexpected values, the example is above for -1 in sqrt
• Stability of variable at its boundary values, for example, when A is 0.000000000000000001
• Data combinations, when A is a string
• Data flow, how variable data changes through the program
• Tables that determine control flow in table-driven code, when A data values could be from table control flow
• Missing code, is above example
• Timing, what if we run a program at midnight
• Compatibility with devices or other software or systems, what if we upgrade the keyboard driver
• Volume or load, A takes the maximal possible value
• Interactions with background tasks, program run while automatic os upgrade also runs
• Side effects of interrupts, your Internet router is blocked, and Gmail opened in browser receives interrupt that network is not available.
• Handling of hardware faults, when the program receives disk drive IO failure (note that this is an exception, not interrupt)
• User interface errors, an example is errors related to CSS
• Compliance with contracts or regulation, when code implements a feature that is not aligned with required regulation
• Whether the software actually delivers the benefits or solves the problems it was intended to address, this is hard to test, because you need to build a working program. The question is: Do we do the right thing?