TL;DR

In this post, we list bug count alternatives that could help you in evaluating software tester’s work. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

It is very tempting to use the reported bug count as a tester work assession method. This could lead to proxy measurement fallacy.

People optimize what we measure them against, at the expense of what we don’t measure.

What alternatives we could use to assess the quality of software tester work?

Documenting Test Work

We do not mean documents in the form of test cases with detailed steps description. We refer here to Session-Based Test Management [Bach] documentation. In that document, tester starts with a charter that describes basic guidelines for a timeboxed session. Here is an example for WordPress application:

Explore Posts using Web Browser to discover that user could access only it’s own Posts.

After we have charters, we write notes with the following tags:

Idea – during exploring, we got an idea about the next charter, testing technique, risk, application feature.

Bug – we think that we found a bug, so we described not in great detail, just enough information that we could later get back to it.

Question – we note our confusion with a question

Opportunity – idea for the next chapter

Fact – we learned something new about the application.

Coaching Other Testers

How tester transfers knowledge to fellow testers?

Bug Reports

Are bug reports written using RIMGEA or some other method?

Regression Testing

How testers samples tests to mitigate regression testing risks?

Variant Bugs

Using idea and opportunity testing note types, so tester explores found bug to find more critical bugs.

Conclusion

We listed ideas on how to better assess the quality of software tester’s work. Remember that people optimize what we measure them against, at the expense of what we don’t measure.