TL;DR

Measuring software quality seems hard at first. The reason is that the software development team is using numbers that are not related to software quality. And the root cause is that we do not know the measurement theory. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

What Is Software Quality

To be able to measure it, we first need to know what is software quality. Heuristic Test Strategy Model [Bach], defines a quality asset of quality criteria (attributes). At the nutshell, the quality criterion is:

A quality criterion is some requirement that defines what the product should be [Bach].

We are back to the chicken/egg problem, what comes first. Many software development teams (including software testers) would probably repeat that they do not have defined requirements, without asking what the root cause of this problem is.

There are requirements, but the project manager either had not bothered to communicate those with the team or had not communicated with the project sponsor to define them.

Quality criterion ends with …ility.

Software quality criteria attributes are known, but your project manager needs to filter them in the PRIORITY list, after discussion with the customer.

Queen option I want it all, and I want it now is not an option, because your project will fail.

Let’s take the Developmentability quality criterion as an example, and it’s subsection, Maintainability. This could be of most interest to the reader because it touches regression testing and refactoring. Maintainability quality is how easy it is to fix, build, or enhance the product.

Measurement

To be able to do the measurement, we need to have:

• the attribute
• the instrument
• the reading
• the measured value (reading on the instrument)
• the metric (instrument scale)

What is our attribute? Let’s pick how EASY it is to FIX the product.

What is the instrument? Let’s pick a stopwatch.

The reading. We measure time from the moment when bug IS CONFIRMED to time when it is fix deployed to production.

The metric. Here come the project manager and team discussion. You want to define a table of bug severity and time to deploy a fix to production:

HIGH: 2 hours

MEDIUM: 2 days

LOW: 2 weeks

Measurement Validity

To determine if this measure useful for the project, we need to determine it’s validity.

We asked the question:

Does this measure how easy it is to fix the product?

If you need more than 2 hours to fix the problem, that could reveal the following development issue:

• we could not replicate an issue
• the developer could not reproduce an issue on its environment
• the developer does not know where in the codebase fix should be made
• the developer does not know how fix affects other parts of the application
• deployment is not automated

Remember

To effectively measure software quality, you need to know what is software quality. Your project manager needs to have a priority list of quality criteria. You need to know what is the measurement, and how to verify is your measurement valid (will do more good than harm to the project).