Category Archives: BBST Foundations

Wrong usage of metric

Reading Time: 2 minutes


This post gives one daily example how metrics could be used in wrong way.

On radio I heard weather condition report: “… and current temperature is missing one degree”. I know that this radio speaker always tries to be comical, but this time he was wrong. Here is why.

BBST Foundations Lecture 6, Introduction to measurement

Measurement is not about counting things, it is about ESTIMATING  the value of something.

Measurement is the empirical (derived from or guided by experience or experiment), objective assignment of numbers to attributes of objects or events (according to a rule derived from a model or theory) with the intent of describing them.

Measurement includes:

  • attribute
  • instrument
  • reading
  • measured value
  • metric => reading on the scale

In the context of weather report, attribute is AIR temperature, instrument is Thermometer, reading is number showed on the instrument, metric is INTERVAL SCALE of Celsius (Celsius because I live in country that uses this scale as de facto a standard for temperature). In United States, I would use Farenheits.

The punchline.

And here comes the punchline, we use interval scale when we do not know/use true zero. In the air temperature contex, on planet Earth, we do not use true zero value ( 0 Kelvins or -273 Celsius) because that value has never been measured as Earth’s air temperature attribute.

Interval scale have its own ZERO value, which is the context of air temperature is value when water freezes. Which is much probable event.

So, saying that we are “missing” one Celsius is wrong, because we do not missing air temperature, air temperature is not missing.

For software testers it is important to understand measurements because tester must be able to give answer to question:

“HOW MUCH testing have we completed and how well have we done it?”

You would not say to your manager: “We are missing one bug”.


Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Scrum testing heuristic: Is feature usable?

Reading Time: 1 minute


This post will describe how I test in project that uses scrum. Tool is Jira and context is early development phase.

We have in JIRA workflow QA phase.

Frontend has code reviews and developers are able to test (there is no automation) what they code using browser.

Here is how I test in described context.

Feature is usable when user can do main flow described in jira card, for example, create business object.

I use browser and Chrome developer tools. Cards have some story description, so I ask for clarification in card comment section. Also, I observe pull request (angular framework) to see which modules changed. I am looking for feature side effects,  when pull request contains changes not directly related to the card.

I approve card when feature is usable. Remember that context is early development phase, and scrum is iterative process, so there is no point to stop the feature if it is usable. For found issues or missing features, I create new card and bug or task, and I connect those two cards.

In BBST foundations lecture 2, you can learn how your testing differs depending on your context.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Feedback on my objection to gherkin language

Reading Time: 2 minutes


In this post I will explain how not to engage in twitter war, but with simple politeness, provoke useful feedback about important topic.

Twitter war is very serious thing. Neil Studd talked about this problem at CAST 2016 lightning talk sessions. During his talk, simple idea popped up in my head. It was time for the experiment.

In my post I constructively object to Gherkin language I expressed my opinion on Gherkin language (which is first part on objection on using UI automation as regression purpose thing).

I always tweet my blog post, and this time I mentioned mr.  George Dinwiddie, because he asked my opinion, also via Twitter.

Here was his first reaction:


He used word strawman, and as I did not know that word, i looked it up:

A straw man is a common form of argument and is an informal fallacy based on giving the impression of refuting an opponent’s argument, while actually refuting an argument that was not advanced by that opponent.[1][Wikipedia]

So I learned new english word, and he objected that I am using gherkin for wrong purpose.

Here was my response:


So I did not engage in twitter war, I politely asked for a blog post. And we have thoughtful response about gherkin language: Using a good tool for the wrong thing. And that was experiment about idea during the Neil lighting talk.

And Ben Simo also responded:


So now we know true purpose of Gherkin language.

And in my original post I forgot to mention how I get information from business people, what product should do so I can create my testing mission. I get them at the beginning of github issues, as several issue comments, written in plain english language, with references to google docs and pictures about the design of the feature.

Using that feedback, here is my follow up on gherkin objection.

We have product examples that could be automate and run so we have immediate evidence that product actually behaves  as this example documents. Which is reasonable purpose.

The problem is:

  • when only those examples are used in testing the product.
  • they are used for regression testing.
  • they are used for UI regression testing (end to end testing) as only automated tests on project

Here is what I suggest:

  • do not stop testing using “manual” (highly intellectual activity)
  • create test mission “using tools at our disposal, lets find out what exactly changed in product from previous release” and do “manual” testing in that part of product. There is no need for automated regression testing on UI level.

As I find this matter highly important for testing community altogether, I would really appreciate your feedback on this matter.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

BBST foundations: time is on my side

Reading Time: 2 minutes

In this blog post you will read about my experience on BBST foundations course. More information about course could be found here. I successfully completed the course.

Why is this better than ISTQB?

I have not taken ISTQB foundations certification exam, but one of my college did and he explained to me his experience. I also skimmed through the exam preparation book. Certification exam consists of 40 multiply choice questions and it is time limited. BBST foundations course also has multiply question quizzes after every lesson, but only for learning purpose. You read lecture readings, watch lecture videos and answer quiz questions using open book technique. Multiply answers are possible. When you found out quiz results, you can discuss and question those answers with teachers and other participants.
One example:
ISQTB: “Give the most RIGHT answer”
BBST: Question has two answers that are true, but by carefully reading the question, you will separate the right answer in the question scope.


All course materials are publicly available. I read all recommended and required readings.

Time is on my side

Course duration is four weeks. Every week has two deadlines. So forget about “I will catch up on weekend.” This time management will not work. Your work must be evenly distributed across whole week. Despite my preparation, in first two weeks I spent 4 hours per day. Later in course, I learned how to better manage my time. This course is based on collaboration with other participants. You will have to provide input in order that your group completes some assignments. Which means that you will have to make some sacrifice. If you are not prepare for that, please do not start this course for the sake of respect towards the other participants that would be ready to make such a sacrifice.

What I learned

This is online course and I have never met with such a learning experience. Something like “Enders Game“. You are doing lesson assignments but actually you are doing your final exam. By taking this course, you will learn how to better manage your limited testing time and will sharpen your skill how to put your thoughts in written form. And “few” things about testing. Good luck!

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

This blog post is about my notes on slide set for BBST Foundations lecture 6: Measurement

Reading Time: 1 minute

This blog post is about my notes on slide set for BBST Foundations lecture 6: Measurement.
In reading Michael Bolton (2007),What Counts?, Michael explains trap in test counting measurement. If you want to take into discussion about measurement in software testing with Michael, you need to read Michael Bolton (2009), “Meaningful Metrics” as a requirement.
Reading Doug Hoffman (2000), “The Darker Side of 
Software Metrics”, explains some popular testing metrics traps. My favourite are side effects of metrics (Wally will buy himself a minivan by writting a lot of bugs).
Cem Kaner and Walter P. Bond (2004), in “Software engineering metrics: What do they measure and how do we know?” answers the question: How do you know that you
are measuring what you think you are measuring?
Erik Simmons (2000), “When Will We Be Done Testing? Software Defect Arrival Modelling with the Weibull Distribution”, gives example of applying Weibull distribution on three different projects.
As a software tester, you have to measure software value to relevant stakeholders.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Notes on slide set for BBST Foundations lecture 5: The impossibility of complete testing.

Reading Time: 1 minute

This blog post is about my notes on slide set for BBST Foundations lecture 5: The impossibility of complete testing.
First read is from Doug Hoffman (2003). “Exhausting your 
test options”. Interesting example how to test sqrt function.
Second reading is from Cem Kaner (1997), The Impossibility of complete testing. We are not quality assurance because we are not authorised to make all moves that will assure quality. We are doing “Good enough testing”.
Third reading is “Factors that influence 
test estimation” by Rex Black. Lists all factors that must be considered when putting out testing effort estimation. “Blog: When do we stop a test?” by Michael Bolton is self descriptive. Cem Kaner (1996), “Negotiating testing  resources: A collaborative approach.” Is about test plan estimations. Mike Kelly, “Estimating testing using 
spreadsheets” gives step by step example how to use spread sheet for testing estimation for Windows Calculator.
Combinatorial mathematics is very important testing tool. It is used to determine testing coverage.
What is complete testing?
1. Run all distinct tests. Test are distinct if they expose bugs that other test will miss.
2. We know that there are no bugs in software.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Notes on slide set for BBST Foundations lecture 4: Programming fundamentals and coverage.

Reading Time: 2 minutes

This blog post is about my notes on slide set for BBST Foundations lecture 4: Programming fundamentals and coverage.
First read: “SOFTWARE NEGLIGENCE AND TESTING COVERAGE” by Cem Kaner. Gives list of 101 coverage measures. Yes, you read it right. There are 101 different coverage measures!
Second reading is “How to misuse code coverage by Brian Marick”. Tool for code coverage is not a hammer, but it can help a lot. Summary: “Developers care more about whether their code does what was intended; product testers care more about whether what was intended is right. Because product testers are looking for requirements-level or specification-level omissions, coverage provides fewer and weaker clues than it does to developers, who are looking for design-level and code-level omissions.”
Third reading is “Got you covered by Michael Bolton”. What is complete testing? All possible test having been perfectly performed. There are known unknowns, and there are unknown unknowns.
Fourth reading is “What every computer scientist should know about floating point arithmetic” by David Goldberg. I skimmed over it. It is detailed paper about floating number format in computers. If your stakeholder business depends in great amount about floating point operations, you will read this paper in detail.
Following reading has interesting title: “An interview with the old man of floating point”.
Sixth reading is “Experience with the cost of different coverage goals for testing”.Gives for which coverage types is reasonable to set 100% coverage using unit testing.
I skimmed that reading. I also skimmed reading book: Charles Petzold (1993), Code: The Hidden Language 
of Computer Hardware and Software. The book explains codes using examples form everyday life. From best friends to modern computers.
The we have: overflow concept, data representation systems (binary, decimal), float, data structures, coverage.
Recommended reading book: “Introduction to software testing”
Interesting interview: I heard for the first time concept of testing requirements that I like very much. It is similar to my notion of common sense requirements.
“””You use coverage analysis to assure quality of your set of tests, not the quality of the actual product.”””

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Notes on slide set for BBST Foundations lecture 3: Oracle.

Reading Time: 1 minute

This blog post is about my notes on slide set for BBST Foundations lecture 3: Oracle.
When I hear world Oracle, first thought that pop up in my mind is Matrix Reloaded scene with Oracle and Neo. Oracle is very important character, and so are testers oracles.
I started with required reading “Testing without a map” by Michael Bolton. Tester sometimes is doing exploratory testing in order to gather information for more detailed test strategy. Oracles (principle) and Heuristics (fallible guides) to Oracles. HICCUPPS heuristics with example.
Second read is “The essence of heuristics” by James Bach. Heuristics give a structure to our testing skill. Heuristics are not best practices.
“Using Heuristic Test Oracles” gave a good example how to use HICCUPPS heuristics.
I found “Definition of Engineering Method” by Koen on following link. Heuristics are important part of engineering method. This should be read by every engineer!
“On testing non-testable programs” lack of oracles is elaborated. Last two manuscripts were produced with typewriter, and it felt like reading ancient manuscripts about secrets of testing.
How we are good at creating and using created oracles, means how are we good at automated testing. I read Michaels blog post about test inputs and expected results. I found an issue in pdf, because although link was properly displayed, when I clicked on it, it pointed me to different blog post. Here is link once again. 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Notes on slide set for BBST Foundations lecture 2: Strategy.

Reading Time: 2 minutes

This blog post is about my notes on slide set for BBST Foundations lecture 2: Strategy.

Before any reading about lecture two, lecture name made me think about word “strategy”. If you want to win, you need strategy in chess, Starcraft, war. Does that imply that testing is game that needs a winner?
And first lecture question is: “Why are we testing”. I looked up in dictionary meaning of word objective because it is important part of answer on previous question.

More brain food: 

  • how we know that test passed or failed?
  • test coverage
  • testing finish
  • where are we between start and finish of testing?
  • how was our testing? Disaster, super tester, useful, missed the point…?

I read lesson required reading: “Managing the 
proportion of testers to (other) developers.”
I have been asked that question several times so far during my testing. And I always answered the number. But, this is only the tip of the iceberg. For start, question is what tasks are testers responsibility? I learned about two factor model (project risk vs project size) for predicting staffing requirements and three factor model (product, project and project process). What is mission of the testing group and how the labour is allocated will help in determining tester/programmer ratio.

Next reading: Heuristic test strategy model.
It has five components, three inputs and one output of software tester work. Big like is “Software is complex and invisible.” Project environment brought up my first testing constraint, and that was time to get software build for J2EE application. While reading about five components I automatically mapped them on my latest project. I think that constrained my learning. I read again thinking about Starcraft 2 game. For example, product function elements in Starcraft 2 context gave me a lot material to think. Mapping heuristic test strategy model to Starcraft 2 game helped me to broader my testing knowledge.

Next reading was Recruiting software testers by Ph.D. Cem Caner. I scanned up to list of questions when hiring test manager. Trying to answer those question, I also learned a lot.

How do I Create Value with my Testing?  also has a list of valuable questions that every tester should ask himself. Doing that, it will become better tester. It also triggered my own ideas.

Should you read recommended reading list? Yes! I scanned only part of Recruiting software tester because that part was for HR personnel. When I will need to create test plan for my next project I will come back to recommended reading list to read it again because it will help me to create better test plan depending on project context.
Also, it will be very helpful to review my self how am I doing on that future testing project.

Conjectures and Refutations: The Growth of Scientific Knowledge (Routledge Classics) is book by Karl Popper. I have met with Popper work in Michael Bolton blog posts and although I do not have access to that book right now (and it is marked with skim reference), I found out part of it as this pdf document. What is all about? Lack of (or poor testing execution) of modern scientific theories. Great read.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Notes on slide set for BBST Foundations lecture 1: Overview and basic definitions.

Reading Time: 1 minute

Today I made a decision and acted on it. I become AST member (95 us$) and enrolled for October BBST Foundation class (125$). I am preparing for that class by reading and watching materials found on following link. In this blog post I will take notes on slide set for BBST Foundations lecture 1: Overview and basic definitions.

My first smile was when I red what typically ask black box and glass box tester (I left to blog reader to conclude which question belongs to which tester).
“Does this do what the users 
(human and software) expect?”

“Does this code do what the 

programmer expects or intends?”

Smile was because this is so simple explanation and I have never came to that explanation by myself. And dear Google, because of the first question, test is not dead.
We have testing approaches (e.g. glass box testing) and testing techniques (e.g. usability testing, dear reader, name any other testing technique).
Distinction between validation (are we building the right it?) and verification (was something implemented correctly?) is now clearer than ever before.
Function is system feature. Functional testing tests one or several features together, on system level. Why parafunctional instead of nonfunctional testing? Because of this: “all the nonfunctional tests are now working…”
Acceptance testing is between interested parties, wether we deliver (internal decision e.g. Google) or wether we accept (e.g. government vs contractor). Independent testing was also defined.
It is important to learn quiz questions and answer exactly what is asked, nothing more. During the quiz questions, candidate will learn new things. This is important tester skill. First challenge on horizon.
Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather