Category Archives: learn testing

Testival #33: How to come up with test idea using lateral thinking?

Reading Time: 4 minutes

TL;DR

After Testival meetup, I always write short summary about what I learned on the meet up. This time learning went in both directions. I gave a talk about lateral thinking as a source for test ideas. I noted audience feedback as part of exercise part of the presentation.

Again, meetup sponsor was Degordian! Thank you.

We had 15 testers and started with usual introduction and what we had tested today confession.

Second talk was about Selenium and it was give by Ana Prpic. It was comprehensive overview  about Selenium history that ended with hands on presentation in webdriver.io against “The Internet” Heroku demo application.

Zeljko gave 5 minute talk about Vim, and I learned that there is vim-adventures game, a fun way to learn Vim keyboard shortcut secrets. Zeljko also provided feature photo for this post. Thanks!

How to come up with test ideas using lateral thinking?

One of the most important software tester skill is to come up with test ideas. There is a number of test techniques that help tester to do this task. One such test technique is lateral thinking.

Lateral thinking is simply put the skill to think outside the box. This talk also had fun part when audience got chance to practice lateral thinking.

I started with question: “How to come up with test ideas?”

Here are the audience answers:

  • acceptance criteria, customer and team
  • explore the application
  • spying on developers!

Last answer is brilliant. On the other side, it can be frightening if you have to spy on you developers to get important information. But in the context of just over listening in “open space” developers conversations, this is brilliant source of test ideas!

Here is list of sources for test ideas:

  • Risk Catalogs

Risk catalog is comprehensive list of risks (the danger of something bad happening [Bach]) in common components or functions that we should test. For example OWASP project have a great collection of risk catalogs in security context. Risk catalogs come in form of checklist, but not in “dangerous” form. Those list will give test ideas, not the actual tests. For example list of Input methods for your application would consist of: mouse, keyboard, unicode with hints that could help you in creating actual tests.

  • Quality Characteristics

Those are quality attributes. For example, security is one quality attribute, and error handling is another one.

  • Test Techniques

This is how testers test. We all know about (do we?) functional testing, but there is many more test techniques.

  • Critical and Lateral Thinking

Critical and Lateral thinking help you to solve problems. There are rather abstract concepts. This presentation is about lateral thinking, and critical thinking is about how to identify and change our assumptions about the problem.

  • Domain Knowledge

Everything you know/do not know about your project.

  • Mnemonics

Mnemonics helps us to memorize a set of heuristics (a fallible  method that COULD help us to solve a problem). SFDPOT is one such mnemonic created by J. Bach.  and it stands for Structure, Function, Data, Platform, Operation and Time. These are all useful sources of test ideas.

  • Mind map collections

Risk list in form of mind map.

  • Tours

Test techniques in form of tours. For example, lets do the claims tour which means let’s use application driven by user documentation.

Test Sphere Card Deck

Ministry of testing card deck that has 100 cards divided in five categories: Patterns, Quality Aspects, Feelings, Techniques and Heuristics.

  • Rubber Duck Testing

When you describe your solution for a problem to a rubber duck. By saying it aloud, you are verifying it and possibly coming up with idea how to refactor your initial solution.

Then I asked another question: “What could you do with a baseball cap?”

Here are the answers, in order they were given:

  • keep head warm
  • head rain protection
  • head cooling aid
  • send a message
  • fashion statement
  • entertainment by throwing it in the air
  • begging aid
  • distance/volume measurement
  • carry object
  • catch object
  • defence
  • camouflage
  • vote collecting
  • small fire estinguisher
  • keep barbecue fire

There was five minutes deadline. There is a pattern, ideas were developing from most common one (what we learned first in our life about a cap) to more complex one (camouflage), that we learned latter in our life.

Lateral thinking is problem solving skill that help us to think out of the box. Shane Snow gives five methods how to spark lateral thinking about the problem:

  1. List the assumptions

Assumption is something that we think is true about the problem, but we do not have proof. Listing those assumptions will help us to test them.

2. Verbalize the convention

Convention is typical approach to solve a problem. How I can solve this problem using convention?

3. Rewrite the question

By rewriting the asked question about the problem, you will get new perspective and ideas.

4. Start backwards

Give a solution to the problem, and test it on given problem.

5. Change perspective

Change user role, change time, change position, and new ideas for the solution will emerge!

For the last 5 minute exercise, I wrote handy mnemonic PAB-RC (Perspective, Assumptions, Backwards, Rewrite, Convention) and presented a problem:

A man is lying dead in a field.
Next to him there is an unopened package.
There is no other creature in the field.
How did he die?

Here are given solutions:

Man fell on his head.

Package fell on his head.

Did not take medicine from the box.

Fell from a plain (this is also change of perspective).

Natural death

Here are perspectives:

Fell from a plain.

Men is in mine field.

Fell from a plain was true, but there was no time to put package in the context.

The answer is: Men is parashooter and in the air parachute fell of his back. Man hit the ground and died.

For exercise, there is a lot of lateral puzzles on the internet. For example, you can try to come up with your own puzzle.

Here is my take on that:

Every working day, around 8.30, our backup server crashes. Why?

References

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How developer should test?

Reading Time: 2 minutes

TL;DR

At one meetup, @neektza, excellent developer, asked me: “Karlo, I learned xy testing framework, but now I do not know which test should I write”? At that time, I could not answer it, but that question stuck with me. As I am almost done reading through resources Exploratory testing pathway provided by Marcel Gehlen, combining that knowledge and my working experience, here is my first draft answer on that question.

How developer should test?

When I speak with developers about new feature, I often get following developer statements:

  • Hmm, this part of feature is complex
  • Oh, this is very fishy

And then comes the punchline, Karlo you should test this risk in following way …

This test should be designed, executed, refactored, executed again by you my dear developer. And yes, you should use Test Driven Development (TDD) technique. And yes, you should automate this experimented (test) in you favourite testing framework.

When developer starts to think about the feature, questioning it, he/she creates test ideas that covers feature risk driven by quality criteria (for example error handling).

The plainest definition of software (formerly known as exploratory) testing is test design and test execution at the same time [James Bach].

Software developer should do the same using TDD.

And I think (you do not need to agree) that developer logical flow is to first write code, then test. Not the vice versa. You first need to have product in order to create experiment (test idea). And TDD is recursive, output of previous test is input for code refactor and new test ideas, same as in software (exploratory) testing.

To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing [James Bach].

The basic rule is this: exploratory testing is called for any time the next test you should perform is not obvious, or when you want to go beyond the obvious. In my experience, that’s most of the time [James Bach].

Software testing difference between software tester and software developer is that developer is doing it against code that it develops, while software tester does it against other product elements (for example with help files, requirements, application, log component,…).

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Be careful with software updates: example for osx text to speech feature

Reading Time: 1 minute

TL;DR

This blog post is about how OSX update affected text to speech feature that I use as  proof reading aid for my blog posts. I will propose a testing charter based on this experience.

As I am not english native speaker, in order to avoid basic grammar errors in blog posts, I always reply blog post using OSX text to speech feature. This proved to be very useful.

In wordpress editor, (html page) I selected blog post (using keyboard) in paragraph chunks, then I clicked right mouse (using touchpad two finger gesture) and selected speech => Start speaking.

No speech.

In order to decide “Do we have a problem here?”, I used oracle comparable product (FEW HICCUPS by M. Bolton), OSX Notes.

Using same steps as in WordPress, there was speech.

Idea, WordPress editor is html/javascript editor, so ttx can not read html?

Idea, what recently changed with my laptop?

Yesterday I upgraded OSX to Sierra version.

Lets check if state of ttx OSX feature (in the end, this is just another program) is affected with upgrade – BBST Test design (by Kaner and Fiedler).

System Preferences => Accessibility => Speech

Oh there is unselected option, “Speak selected text when the key is pressed”. I enabled Command + Z combination.

Go to wordpress editor, select blog post paragraph, hit command + Z, it works!

Try option with context menu described previously, it worked!

We are taking for granted that complex upgrades, such is OSX upgrade, should work out of the box. In this example, ttx speech was not broken, but was set to initial state because of new configuration option.

I described oracle heuristic comparable product (one C from FEW HICCUPS), and initial values  quick test idea.

You can learn about them in BBST Foundations and Test Design courses.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

What I learned at Testival #32 meetup

Reading Time: 2 minutes

TL;DR

This post is about my  Testival #32 meetup learning experience.

Sponsor of this meetup was Degordian, digital agency  where there is no cure for their curiosity 🙂

They recently moved to new location, and this was my first experience of “Apple” style working place. And it was great experience. When looking such premises only through photos, it may give impression of show off.  But every detail in this place is made with reason. The reason is to stimulate your brain for new ideas and new learning experience! Great job!

 

We started with usual introduction of 14 attendants. One newcomer felt like she does not belong there, because of her educational background. But this was good thing, because we had at the meetup several people with diverse educational background. Software testing is not confirming that stream of binary code works/does not work.

Testing is a “meta” activity. It’s not just a task, but a task that generates new tasks (by finding bugs that should be fixed or finding new risks that must be examined). It’s a task that can never be “completed” yet must get “done.” [James Bach]
Testing is an investigation based on, concerned with, or verifiable by observation or experience conducted to provide stakeholders with information about the quality of the product or service under test. [Cem Kaner]

Marko Kruljac from Degordian presented “Integrating Jenkins with your GitHub Pull Request Workflow”.

Marko talks about Jenkins/github workflow.

 

As I have done similar for several clients, I was curious how they cope with that, because Jenkins offers several options through its great Plugin community.

I learned that for github the best option is Github pull request builder. It also made me think how important is to select Jenkins plugin name in order to present what is actually do.

After pizza and beer break, we had three lighting talks.

Zeljko Filipin from Wikimedia presented unicode zero-width non-joiner character. In Ruby interactive console he merged two “emoji” with that character in order to make new one (e.g. man + pan = cook).

Next day I found that with that character you can make very scary things, like publish your own What’s App application in Google play. Read more about that in Gojko Adzic excellent blog post.

Second lighting talk was from Vilim Stubičan from Degordian talked about his learning process (while he was solving Rubik cube with one hand 🙂 ).

I presented the most important software tester tool, a notebook. Read again James and Cem definitions of software testing to know the reason for that fact.

P.S.

During the testing session in preview mode of this post, I found important issue with one photo because it contained Degordian wifi password. Which automation would caught that issue?

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Oracle exercise on real example

Reading Time: 2 minutes

TL;DR

This post is example how to apply oracle heuristic to identify is there a problem. Disclaimer: this blog post is not about some fancy new software testing framework. Pure software testing craft.

You are still here after disclaimer? Great!

Oracles are simply the principle or mechanism by which we recognise a problem. [Ref.]

Please read the article, it is well written and easy to comprehend. Another quality of excellent software tester.

In order to know how to use oracles in software testing, you need to practice. I hope that this example will help you.

I am “forced” to use Microsoft Word in order to create documentation for one project. I decided to insert images from external documentation using “Insert from URL feature”. In that way, when external documentation changes, link would either break or would automatically point to new image.

I clicked in Word Insert menu, then on image icon. After several minutes, i realized that there is no “Insert from URL option”.

I searched with Google to find quick answer:

Go to Insert – Quick Parts – Field…

Then you will get select box with a lot of options, one of them is insert image from URL (why we should bother to put it as first option in the list).

What!? I will repeat that because it sounds like sentence from Monty Python’s Flying Circus sketch:

Go to Insert – Quick Parts – Field…

Hmm…, do we have a problem here? I am calling oracle consistency heuristics Comparable Product into help.

We expect the system to be consistent with systems that are in some way comparable. That might include other products in the same product line, or from the same company. The consistency-with-past-versions (History) heuristic is arguably a special case of this more general heuristic. Competitive products, services, or systems may be comparable in dimensions that could help to discover a problem. Products that are not in the same category but which process the same data (as a word processor might use the contents of a database for a mail merge) are comparable for the purposes of this heuristic. A paper form is comparable with a computerized input form designed to replace it. Indeed, any product with any feature may provide some kind of basis for comparison, whereby someone might recognize a problem or a suggestion for improvement [Ref.].

Lets check Google Docs.

Click Insert menu option. First suboption is Image icon, click on it, there is option window with option By URL. It took me four seconds to find it.

So this word option is not consistent with comparable product because in comparable product is much easier to insert image. Proof that Google docs has better UX than Microsoft Word.

And you can use this as selling pitch for this issue to your product manager.

I once presented oracle consistency heuristics to software testers. Feedback was: Oh, this is fancy and great, but we DO NOT HAVE TIME TO DO THAT!

Then I asked them contra question: How much time you spent in your bug triage sessions?

A lot.

With oracle heuristics, you are first filter for bug. If you can not find inconsistency in listed heuristics, than you will not report this issue. And your bug triage sessions will be much shorter.

 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

UI check automation suggests important project issues

Reading Time: 1 minute

TL;DR

This blog post is my experience about UI test automation applied in various projects.

First disclaimer, this post is not against UI check automation. If not used as a testing hammer, it can help towards better product quality.

How to recognize UI automation as marker for important project issue? If project testing pyramid morphs into testing coan [source: Watirmelon].

  1. skilfull session based testing is replaced with manual repeating of instructions listed in test cases documents.
  2. all automation checks are in UI level, and represent end-to-end checks.

This points to important project issues:

  1. lack of skilfull testing
  2. knowing test automation framework, usually selenium based, is sexier that skilfull testing

What can you do? Start learning and practicing resources listed in point 1. This will help your project to use testing pyramid and help you to fight your desire for ice cream!

 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Exploratory session of Pena Palace

Reading Time: 2 minutes

TL;DR

Using this excellent post by Marcel Gehlen , I am learning about exploratory software testing.  I created github wiki where I put notes about every resource listed in that post. This post is practical exploratory session of Pena palace located in Sintra, Portugal.

My vacation for end of this summer was Portugal tour, organized by Mondo Travel agency. Part of that visit was Pena national palace, located in Sintra town. As I just had read about session based test management by James Bach, I decided to practice it on Pena palace.

# CHARTER

Mission: Cover every walking path allowed to tourists in Pena palace and document interesting parts using Iphone 6s camera.

Note: When you do testing coverage, it is very important to state in report which coverage was done. Cem Kaner listed 101 testing coverage types, so please read it in order to know how complex is test coverage problem. By stating properly your testing mission, it is easier to estimate how much testing sessions is required.

I stated that I would do every walking path allowed to tourists. So no sneaking to restricted areas. Here are some other possible testing coverage types:

  • investigate every wall picture
  • investigate every palace window
  • investigate every palace door
  • investigate every tiles

# START

I know exact time that is when I took first Pena photo.

# TESTER

Karlo Smid

TASK BREAKDOWN

# DURATION

90 minutes

Note: This was “hard” requirement, because if I had exceeded that time, my group would have waited for me.

Timestamp of last picture

# TEST DESIGN AND EXECUTION
90%

# BUG INVESTIGATION AND REPORTING
0%

# SESSION SETUP
10%

As a group, we got info about Pena Palace from our guide. Also, I checked my testing tools, Iphone 6s and Iphone 6s smart battery case.

#CHARTER VS. OPPORTUNITY
80/20

Note: Kitchen looked very interesting. I would have definitely investigate it more if I had had more time.

# DATA FILES

# TEST NOTES

I managed to walk all available paths and document all items of my interest. When you go on vacation that is organized by agency, your are in the group and you need to adapt to given time. This is tradeoff. Positive thing is that you meet new people that have something in common: like to travel!

# BUGS

None

# ISSUES

None

In this post you learned about test coverage and how to apply session based test management during your leisure time. Have you noticed that exploratory word was striked through in TL;DR? James statement is that every testing is exploratory, so there is no need for exploratory word. And I agree with that statement based on my practical experience in last month when I applied my latest knowledge of exploratory testing.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Learning risk management by example

Reading Time: 2 minutes

TL;DR

After taking state of the art software learning courses, I concluded that best way to comprehend knowledge is to learn by using examples of presented  materials.

How to measure quality of a course or workshop? For me, one metric are examples used in course or workshop. Of course, not THE NUMBER of examples, but my subjective measure how examples helped me to understand theory. And what is more important, how those examples help me to remember what I learned.

My examples of such courses are Rapid Software Testing  by Satisfice or BBST courses by AST foundation.

Today is SATURDAY and it was RAINY morning, and I drove to my hometown Zabok in EARLY morning. Security of my trip was jepordised on third stop light. Traffic lights were off. This was intersection with one direction road and intersection had road signs.

Should I wait for traffic lights be repaired? In that case, I will be late.

Since it was SATURDAY, EARLY RAINY morning, the traffic on the intersection was very light and it was intersection with one direction road, I crossed the intersection SAFER than on working day.

Ok, that was example, but what is the topic?

Your project is using a lot of 3rd party software components. Those components could have important security fixes. Deployment of new version for 3rd party component requires testing.

You are at YOUR crossroad with jeopardised security, this time of your product.

Should you upgrade? Will your sprint be late because of it? Do we need to deploy? Can project be secure enough without the patch?

This is security risk analysis for your product in the context of 3rd party component security fix. You need to create questions and answers in the context of your product and related to software security domain.

Can you give us any example?

 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

The Black Swan event

Reading Time: 2 minutes

TL;DR

Yesterday I experienced negative Black Swan event. I will described it along with explanation what is Black Swan event.

This is explained in the book “The Black Swan (Taleb book) which is on my reading list. Every influenced software tester will recommend it.

“””The book focuses on the extreme impact of certain kinds of rare and unpredictable events (outliers) and humans’ tendency to find simplistic explanations for these events retrospectively. This theory has since become known as the black swan theory.”””[Wikipedia]

Yesterday I arrived home with combination of olive and engine oil on my sneakers. The risk: this significantly lowered traction between my shoes and ground, so there was high probability of fall.

I was in grocery store. There was funny smell in the grocery store. Then I heard crashing sound. One lady dropped a bottle of olive oil at the store entrance. I was in a hurry and did not wait for employee to clean this.

A few minutes later in public garage next to the place where I parked there was a car with big oil spill. Car owners were tourists.

My simplistic explanation (thinking about this event for five minutes).

It is highly probable that people drop things in this grocery store because they get sick with funny smell.

Yearly increase of tourist visitors in Zagreb is on average 10%. It is highly probable that you will meet tourist with broken car.

To conclude.

“””The main idea in Taleb’s book is not to attempt to predict Black Swan events, but to build robustness to negative ones that occur and to be able to exploit positive ones. Taleb contends that banks and trading firms are very vulnerable to hazardous Black Swan events and are exposed to losses beyond those that are predicted by their defective financial models.

The book’s position is that a Black Swan event depends on the observer—using a simple example, what may be a Black Swan surprise for a turkey is not a Black Swan surprise for its butcher—hence the objective should be to “avoid being the turkey” by identifying areas of vulnerability in order to “turn the Black Swans white””” [Wikipedia].

My robustness in this context. For last year of so, I pay extra attention to my shoes. They must have Goretex or similar technology, and advanced sole technology. Currently I am using Columbia Men’s Ventfreak Outdry Multisport Mesh Athletic Sneakers with Omni-Grip™ non-marking traction rubber outsole. The reason is my back pain and when I am wearing those shoes, I do not have back pain. By keeping attention to my back problem I also enhanced my robustness to explained Black Swan event.

Hmm, how can I apply that to my software testing?

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

What I learned at Testival #30: learning by playing card games

Reading Time: 2 minutes

TL;DR

This post is about hands on session for learning about software testing using card games.

Intro

At TestBash Brighton2017, on Friday conference day, during the Lean Coffee, one lady ask question that you have probably already heard: “How should I motivate my team of testers?”

I suggested card games. Another participant immediately dismissed me: “I DO NOT LIKE GAMES IN SOFTWARE TESTING BECAUSE THIS IS NOT PROFESSIONAL!”

Noticed that this was his subjective opinion and he did not provide any arguments why is this bad idea. Well dear fellow software tester, this blog post is to prove you wrong.

Introductions

As usually, we started with 2-5 minute introductions. Every tester must be good at explaining things, so this is good way to practice that craft. We had new participants, 3 of them international. Meetup .com is powerful platform.

In this video extract, Priamo explains one concept from excellent book: “The design of everyday things

TestSphere by Ministry of testing

TestSphere is deck of cards about testing concepts, 100 cards divided in 5 dimensions:

  • Heuristics: Possible ways of tackling a problem.
  • Techniques: Clever activities we use in our testing to find possible problems.
  • Feelings: Every feeling that was triggered by your testing should be handled as a fact.
  • Quality Aspects: Possible aspects of your application that may be of interest.
  • Patterns: Patterns in our testing, but also patterns that work against us, while testing such as Biases.

My point of view is that TestSpehere cards can be used for:

  • test plan
  • interview tester
  • share learning

TestSpehere also represents coverage of knowledge that skilfull tester must have. We randomly picked up green cards which represent technique. Each card has:

  • name
  • product level – hint how you should apply this card
  • three examples to seed idea flow

And magic happened! I handed a card to each participant and everybody had something to say about it. Which means we have the knowledge for software testing techniques, and TestShere cards helped us to structure that knowledge.

FluXX

FluXX is card game where rules and goals could be changed on every hand play. You have seven card types:

  • basic rule – draw one, play one. Game initial state
  • new rule – adds rule and removes existing rules if contradicts with them

Hmmm. Are those two rules contradict each other?

  • Goal card – how to win instruction
  • Creeper helps you to lose
  • Keeper helps you to win
  • Action – do something
  • Surprise

Note: play and discard card are two different actions!

How to play? Have one experience player and just play. Start from basic rule, everything else is written on cards (requirements!), but, you will have to think in order to apply them.

Fluxx helps you to practice all types of thinking in fast changing environment.

Game of Set

Helps you to practice multidimensional pattern recognition.

You have four properties: shape, quantity, filling and colour. Each property could have three values. Rule is very simple: you have 12 cards on a table, cards make a set of three if each of four properties are same OR different.

During gameplay I noticed that players usually forgot about one dimension, or they do not take into account that properties could also be DIFFERENT.

Have you ever tested something that has more than four dimensions? Our brain has system one and system two (more about that in Michael Bolton’s Critical thinking for testers.), you need to give system two time to wake up!

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather