Be careful with your testing tools

TL;DR

In this post I will describe how I got different test results using two testing tools Bug Magnet and Counter strings.

Bug Magnet is handy Chrome Extension developed by Gojko Adzic.

It is: “Convenient access to common boundaries and edge cases for exploratory testing.

Counterstings is implementation of James Bach algorithm that quickly reveals character position in string.

In my test, I used them to check maximal number of allowed characters in input form. This test should be simple, right?

One interesting thing happened. I first used Bug Magnet and inserted string with 128 characters. String was accepted and test failed because maximal allowed number should be 64 characters. I did second test with counterstrings. Boundary of 64 characters was successfully detected. What!?

What happened?

For me, any test should provoke sapient thinking process. In modern web applications, input form could be implemented in two ways:

• html 5 standard, where all input constraint checks are done by browser code according to that standard

You can recognize this by inspecting input text box (Chrome developer tools) and if you see something like this:

<form action="/action_page.php">
  Username: <input type="text" name="usrname" maxlength="10"><br>
  <input type="submit" value="Submit">
</form>

then input textbox is implemented in pure html5.

• using one of popular javascript frameworks (Angular for example, and that was the case with application that I was testing).

Then there is javascript code triggered on keyboard typing action. That code usually manipulates DOM tree to create better looking results.

Bug Magnet is implemented in javascript, and it manipulates DOM tree in order to insert the test data. Be careful, because application user does not do that. Application user types directly into input box. Doing that manipulation, Bug Magnet managed to avoid string length check.

Counterstring uses Ruby on Rails code to generate counterstring on server side. It uses javascript to copy/paste counterstring to user clipboard. My input was done with copy/paste, something more closer to application user.