Category Archives: cucumber

Value of UI automation tests

Reading Time: 1 minute

TL;DR

After series of posts about How to kill UI regression testing, maybe you got impression that I am against UI browser automation test scripts. In this post, I will put my thoughts about the value of UI automation tests.

First some context. Here are requirements for excellent UI automation tests:

  • Language that enables you to write less code. Something like Ruby.
  • selenium web driver implementation in that language
  • framework just above selenium webdriver. For example watir-webdriver.
  • Page object pattern implementation
  • test framework, something like rspec. For writing awesome asserts.
  • gherkin implementation. Something like  cucumber. No, I have not changed my mind stated here. But we have different context here.
  • continuous integration tool. Something like jenkins
  • headless browser. Something like xvfb or sauce labs.

What is the value of UI automation test scripts? They represent executable documentation for your product. They help you to answer questions about complicated business scenarios. For example facebook  setting who can contact me.

Your testers will need to have a skill of writing DRY tests using those requirements. So when you have new tester on board, by reading those tests script, he can quickly learn application features.

Those tests would be run on jenkins and run should be triggered manually. Tests will provide you information is some automated feature changed. It is not wise to use those results for you regression testing strategy.

Because those checks, or asserts only check what is coded. Nothing else. And UI is for human users, not for test scripts. So human should always do smart regression testing , using its brain, eyes, ears and fingers.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Feedback on my objection to gherkin language

Reading Time: 2 minutes

TL;DR

In this post I will explain how not to engage in twitter war, but with simple politeness, provoke useful feedback about important topic.

Twitter war is very serious thing. Neil Studd talked about this problem at CAST 2016 lightning talk sessions. During his talk, simple idea popped up in my head. It was time for the experiment.

In my post I constructively object to Gherkin language I expressed my opinion on Gherkin language (which is first part on objection on using UI automation as regression purpose thing).

I always tweet my blog post, and this time I mentioned mr.  George Dinwiddie, because he asked my opinion, also via Twitter.

Here was his first reaction:

screen-shot-2016-09-10-at-6-26-07-pm

He used word strawman, and as I did not know that word, i looked it up:

A straw man is a common form of argument and is an informal fallacy based on giving the impression of refuting an opponent’s argument, while actually refuting an argument that was not advanced by that opponent.[1][Wikipedia]

So I learned new english word, and he objected that I am using gherkin for wrong purpose.

Here was my response:

screen-shot-2016-09-10-at-6-36-01-pm

So I did not engage in twitter war, I politely asked for a blog post. And we have thoughtful response about gherkin language: Using a good tool for the wrong thing. And that was experiment about idea during the Neil lighting talk.

And Ben Simo also responded:

screen-shot-2016-09-10-at-6-40-12-pm

So now we know true purpose of Gherkin language.

And in my original post I forgot to mention how I get information from business people, what product should do so I can create my testing mission. I get them at the beginning of github issues, as several issue comments, written in plain english language, with references to google docs and pictures about the design of the feature.

Using that feedback, here is my follow up on gherkin objection.

We have product examples that could be automate and run so we have immediate evidence that product actually behaves  as this example documents. Which is reasonable purpose.

The problem is:

  • when only those examples are used in testing the product.
  • they are used for regression testing.
  • they are used for UI regression testing (end to end testing) as only automated tests on project

Here is what I suggest:

  • do not stop testing using “manual” (highly intellectual activity)
  • create test mission “using tools at our disposal, lets find out what exactly changed in product from previous release” and do “manual” testing in that part of product. There is no need for automated regression testing on UI level.

As I find this matter highly important for testing community altogether, I would really appreciate your feedback on this matter.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How to prepare cucumber with tables in Ruby

Reading Time: 1 minute

TL;DR

In this post I will explain how to handle cucumber tables in the “Do not repeat yourself [source]” principle.

Data Tables are handy for passing a list of values to a step definition:

Given the following users exist:
 | name | email | twitter |
 | Aslak | aslak@cucumber.io | @aslak_hellesoy |
 | Julien | julien@cucumber.io | @jbpros |
 | Matt | matt@cucumber.io | @mattwynne |

Here is my first take how to index that data that was not DRY:

class UserListPage
 include PageObject
  data_map = {'name => 1, 'email' => 2, 'twitter' => 3}
  div(:name, :id => 'name')
  div(:email, :id => 'email')
  div(:twitter, :id => 'twitter')
end

And using that data_map in step definition:

Given /^the following users exist:$/ do |table|
data = table.raw[1]
data.shift
on UserListPage do |page|
  data.each {|row|
    expect(page.name).to eq row[page.data_map['name']]
    expect(page.email).to eq row[page.data_map['email']]
    expect(page.twitter).to eq row[page.data_map['twitter']]
  end
end

There is no need for data_map attribute, because table object already has this data. Dry version:

Given /^the following users exist:$/ do |table|
data = table.raw[1]
header = table.raw[0]
data.shift
on UserListPage do |page|
  data.each {|row|
    expect(page.name).to eq row[header.index('name')]
    expect(page.email).to eq row[header.index('email')]
    expect(page.twitter).to eq row[header.index('twitter')]
  end
end

And do not forget to remove data_map attribute from page object class.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Be careful with docker upgrade

Reading Time: 1 minute

TL;DR

These days, updating software that we use is very easy. Installation scripts are automated and tested. In this post I will describe the issue caused with docker upgrade and how I analysed issue in order to resolve it.

In one of my previous post, I described how to create docker image for headless selenium chrome testing. And that worked for some time.

Then I needed to upgrade docker from version 1.9.1 to 1.10.2 because of this Docker 1.9.1 hanging at build step “Setting up ca-certificates-java” issue.

After upgrade, Chrome refused to start in docker image.

After I logged in to headless chrome docker container using configuration described in previous post:

>docker-compose run web bash
>google-chrome
>"Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted"

Lets try to downgrade docker. But how to do that? I remembered that every docker machine has docker version:

docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default - virtualbox Running tcp://192.168.99.100:2376 v1.10.2 
dev * virtualbox Running tcp://192.168.99.101:2376 v1.9.1

Which means that with docker upgrade, only default machine is automatically upgraded.

So I switched to dev machine:

eval "$(docker-machine env dev)"

And using docker-compose I was able to run again cucumber headless chrome test:

docker-compose run web /bundle/bin/cucumber features/feature_name

One last issue. docker tool was upgraded, so when you run:

docker ps -a
Error response from daemon: client is newer than server (client API version: 1.22, server API version: 1.21)

Here comes docker version manager to the rescue. Install it and run its help:

dvm help

Conclusion

Docker is complex technology, it consists of great number of moving parts. Be very carefull when you decide to upgrade docker, especially when around early versions like 1.10.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Headless browser testing in docker container

Reading Time: 2 minutes

TL;DR

In this post you will find out how to run headless chrome browser tests on your development machine using prepared and configured docker machine.

Docker enables you to run on your development machine minimal linux machine. You are in control which software will be installed in that linux machine. All linux machine configuration is stored in two files. Docker uses cache, so only changes in machine configuration are rerun.

Install Docker

First, you need to install Docker for your operating system. Fire up docker terminal after you installed it.

Type: docker-machine ls

Output:

Screen Shot 2016-02-14 at 11.11.42 AM

 

docker machine is linux (special docker flavour), that has docker software that will run your docker containers. It runs in virtualbox that is automatically installed with docker.

You can ssh to it:

 

Screen Shot 2016-02-14 at 11.19.15 AM

 

Clone this github repository that contains docker files needed for creating and running docker image. Follow the README instructions. And that it is, you can run headless chrome tests on your development machine. While those tests are running, you can use your development machine for other tasks.

Dockerfile

Here is official Dockerfile reference documentation. Dockerfile is source code for creating docker images. But I would like to save you time that I spent investigating the official documentation, so you will be ready to create your own docker images very quickly.

Important note: in order to be able to create your own docker images, previously you have successfully configured and installed linux, connected to linux machine, and you know your way around in linux shell.

You do not have to create Dockerfile from scratch. There is great Dockerfile repository, called Docker hub. In search box input what you need, e.g headless chrome, and there is high probability that somebody already have prepared what you are looking for.

If this is not the case, you can download Dockerfile and change it according to your needs. Check in Dockerfile reference what every docker command actually do.

docker-compose.yml

This is second file from docker ecosystem. It is a source code for creating docker containers and their interconnections based on docker machines. For example, WordPress site needs (according to good architecture practice) php container and mysql container.

Line 4 in docker-compose.yml maps current directory (your ui automation project folder) to web docker container app folder. That is the magic which enables docker to run your cucumber tests in docker linux container that is ready with headless chrome, something that is not possible on osx.

Volumes are important docker feature, because without them, docker container does not remember any changes.

Additional magic is that bundle install is run on docker-compose build only when Gemfile is changed.

For example:

  1. docker-compose run web bash
  2. cd ~
  3. touch text.txt
  4. exit
  5. docker-compose run web bash
  6. ls -al ~

There is no text.txt file!

Here is composer reference documentation.

Here are instructions how to start testing using headless chrome:

  1. in docker-composer.yml, replace {project_name} with name of your ui automation project folder name.
  2. copy Dockerfile, docker-entrypoint.sh and docker-composer.yml to your ui automation project folder.
  3. start docker terminal, cd to ui automation project folder.
  4. run docker-compose build –force-rm
  5. run your tests: docker-compose run web /bundle/bin/cucumber -f html -o chrome.html features/feature_name.feature

chrome.html file is created in your local folder, in root folder of ui automation repository!

 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How to send shift and special characters in Selenium Webdriver with Ruby

Reading Time: 1 minute

TL;DR

In order to automate one browser scenario, I needed to send ampersand character. It seems that this is not an easy action in selenium webdriver. So, instead testing the web application, I spent some time in order to find out a hack how to send ampersand key to browser in programmatic way.

My first reaction was: Is this for real, selenium-driver ate ampersand character!? Google as helpful, I found following blog post how to resolve this issue. Solution is in Python, so I rewrote it in Ruby.

Conclusion

Instead of trying to find the solution how to automatically send ampersand in ruby to browser, instead I could do that “manually” and continue the real testing activity. So, my dear testers, when you hear developers claiming that they test their application, prove them wrong. They just try to find solutions how to use automation framework, and that has nothing with sapient activity, software testing!

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Tweeting with Firefox is different than tweeting with Chrome

Reading Time: 1 minute

TL;DR

In this post I will explain how simple change in testing environment influenced my testing of simple feature.

Feature is rather simple, tweet something interesting from the application. As this feature is tested, I was creating automated script for it. This feature is important for my client, so I decided that it should be part of regression testing suite.

I developed script for Chrome, and I also decided to check it using Firefox. Remember that in RIMGEA we learned that G stands for Generalize. We need to test our feature by changing the environment, in this case browser itself.

And in Firefox script failed! Why? Because Firefox tweet feature has one extra step in domain of twitter application.

 

firefox_tweet_3

 

It is clear that Firefox requires one extra button click. Since prior to that click, user clicks on “Log in and tweet” button, this is not consistent with that button. User expects that with one click he does two actions, log in to twitter and tweet that item.

chrome_tweet_2

 

 

With this post I amplified on real case how important is to vary your testing environment because in that way you could find interesting issues within the application.

 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How to automate remember me feature using selenium webdriver and cucumber

Reading Time: 2 minutes

TL;DR

 

In this post I will explain problems I encountered and how I resolved them while I was designing automated test for remember me feature.

 

Context of this post is automation of end to end feature that are visible to user via the browser. You will see remember me checkbox on your login page, but that is the point where visibility of this feature ends.

 

How remember me feature actually works?

If you do not check remember me, then when you close browser after you have finished using web application, next time you come back to the application, you will need to login in again.

 

If you check remember me, then when you close browser and come back again to your application, you will be automatically logged in.

Magic behind this feature are cookies. Cookie contains session for your application and information how long that session in cookie is valid. Server side creates the cookie, but browser must act based on the information in that cookie.

 

So when you test remember me feature, you actually test browser and web application features.

 

First issue in test design

 

In order to have faster execution of browser automation tests, I do not close browser between scenarios (Watirmelon first page object anti pattern). But in order to test remember me feature, you need to close the browser. Here is the code that should be put in cucumber env.rb file that resolves this issue:

 

Second issue is scenario run order dependency

 

But that solution introduces another anti pattern, now scenario run from remember me feature file must be always run in same order:

 

 

That means you can not use following cucumber run option:

 

cucumber –order random :5738 remember_me.feature

Here is step that loads cookies:

 

Third issue selenium webdriver cookie domains

In my first implementation, in env.rb file, I just saved all browser cookies to file. But when I loaded those cookies, i got following exception:

 

 

Note: By observing that exception I learned that Firefox webdriver is actually Firefox extension written in javascript. That extension is automatically installed in Firefox before test run.

And here is the line of code that produces exception:

 

 

Observe comment in line 8 🙂 For me it implies that developer did not know how to properly implement cookie domain check. But as it is open source project, we will forgive him.

 

The issue here was that Google Analytics cookies had subdomain:

.app.domain.com

as cookie domain value. As application domain was:

app.domain.com

Google Analytics cookie domain was rejected in firefox webdriver. So before saving cookie values in file, I deleted Google Analytics cookies.

 

Conclusion

 

I am moving this blog from Blogger. While I was publishing this post, I discovered that WordPress needs plugin in order to embed gists to posts. Blogger had that feature out of the post.

 

Finding proper WordPress heuristic: install one that has more stars.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather