TL;DR

This post explains and gives examples for consistency testing oracles. This post raises four problems with this approach. The post is aligned with the Black Box Software Testing Foundations course (BBST) designed by Rebecca Fiedler, Cem Kaner, and James Bach.

Consistency testing oracles are presented by Bach and Bolton. Those are fallible heuristics that could be inconsistent with each other. However, they could also help us with testing ideas.

Let’s examine Mac Disk Utility.

Is a feature consistent within a product?

Capacity is shown as a horizontal colored bar. Is this feature consistent within a product?

Let’s click on container disk1.

Capacity is also shown as a colored bar, every capacity space type has a dedicated color.

We conclude that the capacity bar feature is consistent within disk utility.

Is the feature consistent with the comparable product?

Let’s examine the feature amount of free space. Let’s click on Machintosh HD.

We use as comparable product OsX df utility:

df shows 80Gi as available space. Disk utility shows 93.55GB. Disk Utility is using the SI standard, and df is using the IEC standard. We just conclude that disk utility is not consistent with the df tool.

Is the feature consistent with history?

The current version is Version 17.0.3 (1646). For this consistency, we need to be able to run the previous version of disk utility. And this ability is called testability connected with the project. Then we choose a feature amount of available disk space and check in on the previous version. To have comparable results, the best option is to have both versions on the same laptop.

Is the feature consistent with our image?

Image of an organization that created Disk utility. In this case, Apple. Disk Utility is consistent with this image. UI bar looks and feels, fonts and other UI components feel like produced by Apple (see the third screenshot from the top).

Is the feature consistent with claims?

Where the claims are: documentation, specification, or ads. Let’s check the official announcement for Disk Utility First Aid feature:

From ad is it not obvious which feature is that. On the second screenshot from the top, we can see the First Aid feature. Hmm, it seems that the Apple Marketing team and software developers are using different definitions for the same function? I run First Aid, and Apple can only confirm that disk appears to be ok :).

Is feature consistent with standards or regulations?

Let’s take features of disk size. What is the standard (externally-imposed requirement) for disk size? On screenshot 3, we can see available space in GB. And that is the SI standard.

Is the feature consistent with user expectations?

This is not straight forward, because we need to figure out who are users of disk utility. This could be both technical and non-technical users. For professional users, disk utility fulfills all their expectations. But what about none technical users? Let’s say that nontechnical user wants to know the amount of available disk space. On screenshot 4, there is total and open disk space, shown on the UI bar. UI bar is prominent because it depicts available space information. None technical user can say: Oh, 2/3 of my disk is full after 4 years of usage. Its time for disk housekeeping!

Consistent with purpose.

The purpose of disk utility is to manage Macbook disk. First aid, amount of free space, managing external storage, adding new partitions. Those all features are aligned with the purpose of a disk utility tool.

Conclusion

Doing consistency heuristics on any application is testing, and it is a fun activity!