TL;DR

This post is a practical example of comparable product oracle usage. This is one oracle from original HICCUPPS mnemonic created by Bach and Bolton.

When you learn something new from the software testing domain, the best way to show that you understand the topic is to use it in your daily job. I recently switched from Spacemacs Emacs to Visual Studio Code editor. Here is how I tested the Visual Studio Code search feature.

Search Feature

A good tester first consults the documentation. Menu option Help, and in the Search input box, I entered “search.” This is what I got:

Tool pointed me to View menu option Search, which starts the actual search feature. Nothing about feature documentation.

No luck with web documentation either. When I entered in docs search “search” text, I got articles not related to the search:

An unskilled tester would use the power word “Let’s do exploratory testing” on Search feature. Which is basically software testing, doing a set of small experiments to learn by ourselves how Search feature actually works.

We have following feature functions, organized in two rows:

First row buttons, from left to right: Refresh, Clear Search Results, Collapse All.

Second-row buttons, from left to right: Search input box with placeholder text Search, Match Case, Match Whole Word, Use Regular Expression.

Let’s test the search input box. For comparable products, we will use grep run from zsh shell Terminal in visual studio.

We search for the helper method:

insert_user()

The first experiment: search for inser_user() by clicking on the Match Case button.

In Terminal, run following grep cmd:

grep -H -r 'insert_user()' *

Which recursively searches all files from the current folder for insert_user(), where each output line contains file path and found text.

We helped ourselves to consolidate Visual Studio Code search results with Collapse All button.

Visual Studio Code: 62 results in 10 files

Grep: 47 results in 6 files.

Let’s investigate the difference. The first result, seeds.exs was not found with grep, let’s expand this result to see what was actually found. Note, we helped ourselves by increasing the width of search results:

It seems that Visual Studio Code included results as the input search string was a regular expression. But how? We only clicked on the Match Case button. Note on the First experiment result screenshot that we have a blue box around Match Whole Word and Use Regular expression. And in Visual Studio Code Search Feature screenshot, all three buttons are with a blue box. This is because, by default, all three features are on. By clicking on Match Case we actually turned off that feature. Match Case did not make any difference in the search results because I did not use mix cases for insert_user() methods.

So we actually used regular expression features. Lets click on it, which actually turns it off:

Bingo! Same results as with grep.

Lets user regular expression with grep where we escaped ( and ), making them regular expression grouping characters:

grep -H -r 'insert_user\(\)' * | grep -v _build | grep -v deps | grep -v cover

Grep found seeds.exs this time. We excluded _build, cover, and deps folders. Those are automatically excluded in Visual Studio Code because those are listed in .gitignore and automatically excluded in Visual Studio Code. Smart feature.

Conclusion

There was no documentation for the search feature. We tested the search feature and used comparable product grep as a comparable oracle. What was confusing was the fact that setting features on/off by clicking on it, also triggered actual search action. My failed heuristic was that only a search is happening on that click.