Wrong usage of metric

Reading Time: 2 minutes


This post gives one daily example how metrics could be used in wrong way.

On radio I heard weather condition report: “… and current temperature is missing one degree”. I know that this radio speaker always tries to be comical, but this time he was wrong. Here is why.

BBST Foundations Lecture 6, Introduction to measurement

Measurement is not about counting things, it is about ESTIMATING  the value of something.

Measurement is the empirical (derived from or guided by experience or experiment), objective assignment of numbers to attributes of objects or events (according to a rule derived from a model or theory) with the intent of describing them.

Measurement includes:

  • attribute
  • instrument
  • reading
  • measured value
  • metric => reading on the scale

In the context of weather report, attribute is AIR temperature, instrument is Thermometer, reading is number showed on the instrument, metric is INTERVAL SCALE of Celsius (Celsius because I live in country that uses this scale as de facto a standard for temperature). In United States, I would use Farenheits.

The punchline.

And here comes the punchline, we use interval scale when we do not know/use true zero. In the air temperature contex, on planet Earth, we do not use true zero value ( 0 Kelvins or -273 Celsius) because that value has never been measured as Earth’s air temperature attribute.

Interval scale have its own ZERO value, which is the context of air temperature is value when water freezes. Which is much probable event.

So, saying that we are “missing” one Celsius is wrong, because we do not missing air temperature, air temperature is not missing.

For software testers it is important to understand measurements because tester must be able to give answer to question:

“HOW MUCH testing have we completed and how well have we done it?”

You would not say to your manager: “We are missing one bug”.


Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Testival #33: How to come up with test idea using lateral thinking?

Reading Time: 4 minutes


After Testival meetup, I always write short summary about what I learned on the meet up. This time learning went in both directions. I gave a talk about lateral thinking as a source for test ideas. I noted audience feedback as part of exercise part of the presentation.

Again, meetup sponsor was Degordian! Thank you.

We had 15 testers and started with usual introduction and what we had tested today confession.

Second talk was about Selenium and it was give by Ana Prpic. It was comprehensive overview  about Selenium history that ended with hands on presentation in webdriver.io against “The Internet” Heroku demo application.

Zeljko gave 5 minute talk about Vim, and I learned that there is vim-adventures game, a fun way to learn Vim keyboard shortcut secrets. Zeljko also provided feature photo for this post. Thanks!

How to come up with test ideas using lateral thinking?

One of the most important software tester skill is to come up with test ideas. There is a number of test techniques that help tester to do this task. One such test technique is lateral thinking.

Lateral thinking is simply put the skill to think outside the box. This talk also had fun part when audience got chance to practice lateral thinking.

I started with question: “How to come up with test ideas?”

Here are the audience answers:

  • acceptance criteria, customer and team
  • explore the application
  • spying on developers!

Last answer is brilliant. On the other side, it can be frightening if you have to spy on you developers to get important information. But in the context of just over listening in “open space” developers conversations, this is brilliant source of test ideas!

Here is list of sources for test ideas:

  • Risk Catalogs

Risk catalog is comprehensive list of risks (the danger of something bad happening [Bach]) in common components or functions that we should test. For example OWASP project have a great collection of risk catalogs in security context. Risk catalogs come in form of checklist, but not in “dangerous” form. Those list will give test ideas, not the actual tests. For example list of Input methods for your application would consist of: mouse, keyboard, unicode with hints that could help you in creating actual tests.

  • Quality Characteristics

Those are quality attributes. For example, security is one quality attribute, and error handling is another one.

  • Test Techniques

This is how testers test. We all know about (do we?) functional testing, but there is many more test techniques.

  • Critical and Lateral Thinking

Critical and Lateral thinking help you to solve problems. There are rather abstract concepts. This presentation is about lateral thinking, and critical thinking is about how to identify and change our assumptions about the problem.

  • Domain Knowledge

Everything you know/do not know about your project.

  • Mnemonics

Mnemonics helps us to memorize a set of heuristics (a fallible  method that COULD help us to solve a problem). SFDPOT is one such mnemonic created by J. Bach.  and it stands for Structure, Function, Data, Platform, Operation and Time. These are all useful sources of test ideas.

  • Mind map collections

Risk list in form of mind map.

  • Tours

Test techniques in form of tours. For example, lets do the claims tour which means let’s use application driven by user documentation.

Test Sphere Card Deck

Ministry of testing card deck that has 100 cards divided in five categories: Patterns, Quality Aspects, Feelings, Techniques and Heuristics.

  • Rubber Duck Testing

When you describe your solution for a problem to a rubber duck. By saying it aloud, you are verifying it and possibly coming up with idea how to refactor your initial solution.

Then I asked another question: “What could you do with a baseball cap?”

Here are the answers, in order they were given:

  • keep head warm
  • head rain protection
  • head cooling aid
  • send a message
  • fashion statement
  • entertainment by throwing it in the air
  • begging aid
  • distance/volume measurement
  • carry object
  • catch object
  • defence
  • camouflage
  • vote collecting
  • small fire estinguisher
  • keep barbecue fire

There was five minutes deadline. There is a pattern, ideas were developing from most common one (what we learned first in our life about a cap) to more complex one (camouflage), that we learned latter in our life.

Lateral thinking is problem solving skill that help us to think out of the box. Shane Snow gives five methods how to spark lateral thinking about the problem:

  1. List the assumptions

Assumption is something that we think is true about the problem, but we do not have proof. Listing those assumptions will help us to test them.

2. Verbalize the convention

Convention is typical approach to solve a problem. How I can solve this problem using convention?

3. Rewrite the question

By rewriting the asked question about the problem, you will get new perspective and ideas.

4. Start backwards

Give a solution to the problem, and test it on given problem.

5. Change perspective

Change user role, change time, change position, and new ideas for the solution will emerge!

For the last 5 minute exercise, I wrote handy mnemonic PAB-RC (Perspective, Assumptions, Backwards, Rewrite, Convention) and presented a problem:

A man is lying dead in a field.
Next to him there is an unopened package.
There is no other creature in the field.
How did he die?

Here are given solutions:

Man fell on his head.

Package fell on his head.

Did not take medicine from the box.

Fell from a plain (this is also change of perspective).

Natural death

Here are perspectives:

Fell from a plain.

Men is in mine field.

Fell from a plain was true, but there was no time to put package in the context.

The answer is: Men is parashooter and in the air parachute fell of his back. Man hit the ground and died.

For exercise, there is a lot of lateral puzzles on the internet. For example, you can try to come up with your own puzzle.

Here is my take on that:

Every working day, around 8.30, our backup server crashes. Why?


Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How developer should test?

Reading Time: 2 minutes


At one meetup, @neektza, excellent developer, asked me: “Karlo, I learned xy testing framework, but now I do not know which test should I write”? At that time, I could not answer it, but that question stuck with me. As I am almost done reading through resources Exploratory testing pathway provided by Marcel Gehlen, combining that knowledge and my working experience, here is my first draft answer on that question.

How developer should test?

When I speak with developers about new feature, I often get following developer statements:

  • Hmm, this part of feature is complex
  • Oh, this is very fishy

And then comes the punchline, Karlo you should test this risk in following way …

This test should be designed, executed, refactored, executed again by you my dear developer. And yes, you should use Test Driven Development (TDD) technique. And yes, you should automate this experimented (test) in you favourite testing framework.

When developer starts to think about the feature, questioning it, he/she creates test ideas that covers feature risk driven by quality criteria (for example error handling).

The plainest definition of software (formerly known as exploratory) testing is test design and test execution at the same time [James Bach].

Software developer should do the same using TDD.

And I think (you do not need to agree) that developer logical flow is to first write code, then test. Not the vice versa. You first need to have product in order to create experiment (test idea). And TDD is recursive, output of previous test is input for code refactor and new test ideas, same as in software (exploratory) testing.

To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing [James Bach].

The basic rule is this: exploratory testing is called for any time the next test you should perform is not obvious, or when you want to go beyond the obvious. In my experience, that’s most of the time [James Bach].

Software testing difference between software tester and software developer is that developer is doing it against code that it develops, while software tester does it against other product elements (for example with help files, requirements, application, log component,…).

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Ruby on Rails bottom up security – sensitive data exposure

Reading Time: 1 minute


This post is about risk sensitive data exposure in your Ruby on Rails application. It will cover unauthorized access and cross site request forgery check (CSRF).

Unauthorized access risk is simple. User can access data that it is not supposed to access. Here you need to check source code of every controller. Your job is easier if developer named controllers by their purpose. For example:




make a big difference.

Rails application is using controller filters, for example:


If controller is public, then there would not be any before_action filter authorization method. If authorization is required, then there would be  before_action filter, for example, auth_user or auth_admin, depending on controller context.

And this is perfect candidate for automatization code. Developer should write simple tests with call for every controller, and result checks of HTTP status code that should be 403 for authorized controllers, and 200 for public one.

If there is role access, check authorization controllers with appropriate role credentials.

Cross site request forgery is when hacker tricks user to execute in his browser http request that modifies data (PUT, POST, DELETE). Ruby has out of the box CSRF protection, that adds additional hidden token parameter in all such requests. Of course, that protection could be turned off.

You should search your code base for:

grep -H -r 'protect_from_forgery except:' * | less

and you should discuss with developer do you really need that exception.


Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Be careful with software updates: example for osx text to speech feature

Reading Time: 1 minute


This blog post is about how OSX update affected text to speech feature that I use as  proof reading aid for my blog posts. I will propose a testing charter based on this experience.

As I am not english native speaker, in order to avoid basic grammar errors in blog posts, I always reply blog post using OSX text to speech feature. This proved to be very useful.

In wordpress editor, (html page) I selected blog post (using keyboard) in paragraph chunks, then I clicked right mouse (using touchpad two finger gesture) and selected speech => Start speaking.

No speech.

In order to decide “Do we have a problem here?”, I used oracle comparable product (FEW HICCUPS by M. Bolton), OSX Notes.

Using same steps as in WordPress, there was speech.

Idea, WordPress editor is html/javascript editor, so ttx can not read html?

Idea, what recently changed with my laptop?

Yesterday I upgraded OSX to Sierra version.

Lets check if state of ttx OSX feature (in the end, this is just another program) is affected with upgrade – BBST Test design (by Kaner and Fiedler).

System Preferences => Accessibility => Speech

Oh there is unselected option, “Speak selected text when the key is pressed”. I enabled Command + Z combination.

Go to wordpress editor, select blog post paragraph, hit command + Z, it works!

Try option with context menu described previously, it worked!

We are taking for granted that complex upgrades, such is OSX upgrade, should work out of the box. In this example, ttx speech was not broken, but was set to initial state because of new configuration option.

I described oracle heuristic comparable product (one C from FEW HICCUPS), and initial values  quick test idea.

You can learn about them in BBST Foundations and Test Design courses.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

What I learned at Testival #32 meetup

Reading Time: 2 minutes


This post is about my  Testival #32 meetup learning experience.

Sponsor of this meetup was Degordian, digital agency  where there is no cure for their curiosity 🙂

They recently moved to new location, and this was my first experience of “Apple” style working place. And it was great experience. When looking such premises only through photos, it may give impression of show off.  But every detail in this place is made with reason. The reason is to stimulate your brain for new ideas and new learning experience! Great job!


We started with usual introduction of 14 attendants. One newcomer felt like she does not belong there, because of her educational background. But this was good thing, because we had at the meetup several people with diverse educational background. Software testing is not confirming that stream of binary code works/does not work.

Testing is a “meta” activity. It’s not just a task, but a task that generates new tasks (by finding bugs that should be fixed or finding new risks that must be examined). It’s a task that can never be “completed” yet must get “done.” [James Bach]
Testing is an investigation based on, concerned with, or verifiable by observation or experience conducted to provide stakeholders with information about the quality of the product or service under test. [Cem Kaner]

Marko Kruljac from Degordian presented “Integrating Jenkins with your GitHub Pull Request Workflow”.

Marko talks about Jenkins/github workflow.


As I have done similar for several clients, I was curious how they cope with that, because Jenkins offers several options through its great Plugin community.

I learned that for github the best option is Github pull request builder. It also made me think how important is to select Jenkins plugin name in order to present what is actually do.

After pizza and beer break, we had three lighting talks.

Zeljko Filipin from Wikimedia presented unicode zero-width non-joiner character. In Ruby interactive console he merged two “emoji” with that character in order to make new one (e.g. man + pan = cook).

Next day I found that with that character you can make very scary things, like publish your own What’s App application in Google play. Read more about that in Gojko Adzic excellent blog post.

Second lighting talk was from Vilim Stubičan from Degordian talked about his learning process (while he was solving Rubik cube with one hand 🙂 ).

I presented the most important software tester tool, a notebook. Read again James and Cem definitions of software testing to know the reason for that fact.


During the testing session in preview mode of this post, I found important issue with one photo because it contained Degordian wifi password. Which automation would caught that issue?

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Ruby on Rails bottom up security – mass assignment

Reading Time: 2 minutes


Mass assignment is security risk where user can create/update data attributes that is not allowed to update.

Here is an example. Imagine application that registers your employees working hours. When user logs in it sets start time, and when it logs out it sets end time. Pretty simple feature.

User login form has username/password input fields. Imagine that user can temper its login timestamp using login request. How? Your employee friend is skilfull tester, and he knows how to send POST request using Postman tool. Using Chrome developer tools he/she finds out the login attributes and now he tries to guess login timestamp attribute:

  • createtime
  • logintime
  • login_time
  • create_time, …

Those names set with date values in the past (he/she wants his friend to work less) are sent using curl (no need to know cookie!). Heuristic to know when correct time attribute is guessed is very simple. There is another url endpoint, or even login response, that will return login time.

How is that possible!? Ruby on Rails got its popularity because is has a lot of default features that made developer work much easier. One of those features is to automatically accept all http request input parameters that match to available ORM (object relational model) object attributes.

Creation timestamp is an example of attribute that should be set by application, not the user input.

Remember, never trust the user input. And hacker loves default framework features.

Ruby on Rails in current version does not allow mass assignment. Every input parameter must be listed in


method in order to be accepted by ORM.

Using super power tool grep, you should search your Rails codebase for this:

grep -H -r 'permit' * | less

Using your knowledge about the application, you need to conclude (look ma, no automation here!) are those parameter allowed to be listed in permit method in the first place.

I also strongly advise communication with your developers 🙂 in order to make the decision.


Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Oracle exercise on real example

Reading Time: 2 minutes


This post is example how to apply oracle heuristic to identify is there a problem. Disclaimer: this blog post is not about some fancy new software testing framework. Pure software testing craft.

You are still here after disclaimer? Great!

Oracles are simply the principle or mechanism by which we recognise a problem. [Ref.]

Please read the article, it is well written and easy to comprehend. Another quality of excellent software tester.

In order to know how to use oracles in software testing, you need to practice. I hope that this example will help you.

I am “forced” to use Microsoft Word in order to create documentation for one project. I decided to insert images from external documentation using “Insert from URL feature”. In that way, when external documentation changes, link would either break or would automatically point to new image.

I clicked in Word Insert menu, then on image icon. After several minutes, i realized that there is no “Insert from URL option”.

I searched with Google to find quick answer:

Go to Insert – Quick Parts – Field…

Then you will get select box with a lot of options, one of them is insert image from URL (why we should bother to put it as first option in the list).

What!? I will repeat that because it sounds like sentence from Monty Python’s Flying Circus sketch:

Go to Insert – Quick Parts – Field…

Hmm…, do we have a problem here? I am calling oracle consistency heuristics Comparable Product into help.

We expect the system to be consistent with systems that are in some way comparable. That might include other products in the same product line, or from the same company. The consistency-with-past-versions (History) heuristic is arguably a special case of this more general heuristic. Competitive products, services, or systems may be comparable in dimensions that could help to discover a problem. Products that are not in the same category but which process the same data (as a word processor might use the contents of a database for a mail merge) are comparable for the purposes of this heuristic. A paper form is comparable with a computerized input form designed to replace it. Indeed, any product with any feature may provide some kind of basis for comparison, whereby someone might recognize a problem or a suggestion for improvement [Ref.].

Lets check Google Docs.

Click Insert menu option. First suboption is Image icon, click on it, there is option window with option By URL. It took me four seconds to find it.

So this word option is not consistent with comparable product because in comparable product is much easier to insert image. Proof that Google docs has better UX than Microsoft Word.

And you can use this as selling pitch for this issue to your product manager.

I once presented oracle consistency heuristics to software testers. Feedback was: Oh, this is fancy and great, but we DO NOT HAVE TIME TO DO THAT!

Then I asked them contra question: How much time you spent in your bug triage sessions?

A lot.

With oracle heuristics, you are first filter for bug. If you can not find inconsistency in listed heuristics, than you will not report this issue. And your bug triage sessions will be much shorter.


Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

UI check automation suggests important project issues

Reading Time: 1 minute


This blog post is my experience about UI test automation applied in various projects.

First disclaimer, this post is not against UI check automation. If not used as a testing hammer, it can help towards better product quality.

How to recognize UI automation as marker for important project issue? If project testing pyramid morphs into testing coan [source: Watirmelon].

  1. skilfull session based testing is replaced with manual repeating of instructions listed in test cases documents.
  2. all automation checks are in UI level, and represent end-to-end checks.

This points to important project issues:

  1. lack of skilfull testing
  2. knowing test automation framework, usually selenium based, is sexier that skilfull testing

What can you do? Start learning and practicing resources listed in point 1. This will help your project to use testing pyramid and help you to fight your desire for ice cream!


Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Ruby on Rails bottom up security – Cross site scripting (XSS) check

Reading Time: 1 minute


This post explains how to check your Rails application source code for cross site scripting (XSS) attack.

Cross site scripting means that your application accepts html code as user input. Biggest issue is <script> tag, that allows user to execute javascript code in the context of your application. Second one is <img> tag that can also be used for code execution.

Rails by default escapes all input, which means that html code will be transformed, so browser will not interpret it as html:

<script>alert("Session based test management");</script>` => `&lt;script&gt;alert(&quot;Session based test management&quot;);&lt;/script&gt;

But some applications, such as github, allow users to have text formatting options.

Dirty way is to allow html input (github is using markdown language), and Rails have api methods for that:



This is ok as long there is not direct user input as parameter of that method (for your editor implementation, you also want to use markdown). Never trust your users!

Use this for code check:

grep -H -r 'html_safe' * | less

grep -H -r 'raw' * | less

There is one more important xss security attack vector. When you open link in new tab, application from that new tab can control, using javascript, application in original tab.

Use this for source code check:

grep -H -r 'target="_blank"' * | less

and make sure that link tag also has this option:



Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Blog that makes software testing interesting and exciting.