TL;DR

This post explains combinations. The most popular usage of combination in daily life is the lottery. Let see what lottery and software testing have in common. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

To test everything, we would have to test every possible combination of input to every possible combination of a variable. As this number of combinations becomes quickly significant for a number of inputs and variables, there are sampling techniques that could help us reduce that number.

As tester’s, it is highly probable that we would first encounter combinations with configuration settings. For example, in web applications, we use browsers. Here are some configuration browser variables:

• vendor
• version
• operating system
• operating system version
• resolution
• headless mode
• incognito mode
• memory amount

The app could work in Chrome, IE, Firefox on Windows. But the combination Chrome Canary on Windows 7 with 2G RAM in headless incognito mode could crash Chrome.

The Basic Combination Rules

We have two dices. When we throw them at the same time, we have 36 possible outcomes of that throw. Each dice has six numbers. We have two dices. A total number of throw combinations is:

6 x 6 = 36

What is the number of combinations if we flip two coins at the same time?

How To Sample?

The easiest way is to dismiss variable or variable values with the lowest risk. For example, reject the Opera browser, of the Windows 7 operating system. We have a low probability that the user will use that combination. But there is also “Magic.”

All Pairs Sampling

The idea is as follows:

The most common bugs in a program are generally triggered by either a single input parameter or an interaction between pairs of parameters [Wikipedia].

So All Pair algorithm [example] would sample from all possible combinations only those combinations where variables change the value in pairs. As this is not easy to comprehend, let’s try the example. There is a PairWise online tool. Here I created a simple model with three variables with three possible values. Click on Generate Pairwise and Generate All Combinations. We got from 27 combinations, a Pairwise sample of 9 combinations. How does this work? Open All Combinations and take the first row:

python, windows, QA

This row is also present in the Pairwise table. But following two:

python, windows, dev

python, windows, DevOps

are not because they differ from the first row only in the value of ONE VARIABLE.

This is why the next row in the PairWise table is:

python, Linux, dev

because it differs from the first row in the PairWise table in values of TWO VARIABLES.

Heuristics is that there is a HIGHER probability of finding bugs when we change input variables values in PAIRS. Note that this heuristic could fail you, and you could still miss a bug.