Category Archives: UI regression

Value of UI automation tests

Reading Time: 1 minute

TL;DR

After series of posts about How to kill UI regression testing, maybe you got impression that I am against UI browser automation test scripts. In this post, I will put my thoughts about the value of UI automation tests.

First some context. Here are requirements for excellent UI automation tests:

  • Language that enables you to write less code. Something like Ruby.
  • selenium web driver implementation in that language
  • framework just above selenium webdriver. For example watir-webdriver.
  • Page object pattern implementation
  • test framework, something like rspec. For writing awesome asserts.
  • gherkin implementation. Something like  cucumber. No, I have not changed my mind stated here. But we have different context here.
  • continuous integration tool. Something like jenkins
  • headless browser. Something like xvfb or sauce labs.

What is the value of UI automation test scripts? They represent executable documentation for your product. They help you to answer questions about complicated business scenarios. For example facebook  setting who can contact me.

Your testers will need to have a skill of writing DRY tests using those requirements. So when you have new tester on board, by reading those tests script, he can quickly learn application features.

Those tests would be run on jenkins and run should be triggered manually. Tests will provide you information is some automated feature changed. It is not wise to use those results for you regression testing strategy.

Because those checks, or asserts only check what is coded. Nothing else. And UI is for human users, not for test scripts. So human should always do smart regression testing , using its brain, eyes, ears and fingers.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How to kill UI regression testing: continuous integration

Reading Time: 2 minutes

TL;DR

Continuous integration (CI) is last element that successfully kills your UI regression testing. In this post I will explain how CI system should fit in software development process.

There is always lively debate what should be automated in software development process. Pipeline elements of continuous integration should definitely be automated.

What are elements of pipeline continuous integration? Every command of operating system that you run as software tester is that element. And software tester usually run several of those commands in a pipeline as their daily task. Those os commands should be put in a script, and that script should become running element of your continuous integration tool (e.g. Jenkins).

Script can be Dockerfile or bash shell script. It depends on the technology that you use. The goal is to have one click button in your CI tool that will do following:

  • get the latest code from git branch (e.g. integration)
  • run development framework tests. Whole suite must run in less than 10 minutes. Otherwise, these are not development framework tests
  • deploy the application
  • run database migration and fixture files (hopefully, your development framework offers those features out of the box. Otherwise, find appropriate tool.)

Whole process must be done in less that 10 minutes.

Now you are ready to do you regression test strategy that includes:

  • observing changed files from last deploy using git pull requests and commits
  • investigating development framework test results, including test coverage report (be sure to know what that coverage means)
  • decide which end to end tests should be executed.

That concludes How to kill UI regression testing series. You will probably say, but I can not implement that in my project. That is ok, but now you know where should your project streamline in order to have better quality.

You have measure points for your project that can you use when you are asked:

  • Why we missed that critical bug?
  • How can we improve quality of our project?

On the other side, you can use this information in your interview for testing position or you can get information about the quality of a project that you are starting with.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How to kill UI regression testing: git source control

Reading Time: 1 minute

TL;DR

This is third part in series how to kill UI regression testing. It is about git source control system and its web application part, github, bitbucket or gitlab. I have already wrote about this topic in Product moving parts as a source for test strategy, and Micro tracking changes for regression test. In this post, I will extend my thoughts from those two posts.

So git helps you to determine which part of application feature could be affected by the code change. Precondition is that you need to be able to read the code. Just read and understand functionality in that code. Many software testers started in software testing because their are afraid of code. I think that to be able to read the code could be great booster for software tester career.  And by reading pull request, you have very valuable mentor, the author of code himself.

Soon, you will find different code change patterns. For example, change in function signature in helper module could break more features than change in one module method. You can learn about that very quickly, the most important is that you should stop be afraid of reading the code.

And basically, you are doing code review. You will have questions and by asking them you could find issues before code is deployed and put in action.

Next post will be about Jenkins as an example of continuous integration tool.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How to kill UI regression testing: development framework tests

Reading Time: 2 minutes

TL;DR

This is second post about idea how to kill UI regression testing. In first post I described symptoms for UI regression testing that I would like to kill. I will describe development framework tests that are precondition for killing UI regression testing.

Development framework tests (DEFRATE) are automated tests that are written by developers. Here are reasons why I think that should be the case for every development project.

You have a developer that is a “rock star” in some development framework, lets take for example Ruby on Rails (you can replace that with development framework on your project). Lets see what Ruby on Rails framework offers in testing front.

Here is official documentation on testing Ruby on Rails application. Here is additional ruby testing toolbox. Every software developer that claims Ruby on Rails rockstar title, should be master in any of those ruby testing frameworks.

But which tests should developer create?

Developer test should create tests in DEFRATE that CHECK their code design.

Developer should check all flows that he created in its mind. Those code flows must PAST through the RUNTIME (lets say processor instructions). Tests should follow DEFRATE practice and use all features. For example, developer wrote some code that communicates with database. Essential part of test should be code that mocks database. Good DEFRATE enables easy mock code creation. And when somebody changes code that is mocked in a test, mocked code will fail and detect REGRESSION in code. Lowest and most quickest way to detect regression change.

Side effect of DEFRATE tests are tools that enables you to have code coverage by running those tests. Code coverage information is excellent source of information for you test strategy on UI level. And remember, code line coverage is just one testing coverage out of 101 testing coverage metrics. Here is excellent resource about testing coverage [Kaner: Software Negligence and Testing Coverage]

Google testing blog is also excellent resource for DEVELOPERS how to write good DEFRATE tests.

And the best way to learn testing framework is to read a book about it and do ALL the exercises listed in a book.

But my developers do not write DEFARTE tests! So I must stick with my UI regression testing. Ok, but now you know metrics that enables you to provide information about software quality in your project. And you have directions how to make quality in your project much better.

In next post, I will discuss source control tools that can help you to kill UI regression testing.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How to kill UI regression testing: the simptoms

Reading Time: 2 minutes

TL;DR

This is first post in what should be series of post. Idea is: How to kill UI regression testing. Idea started at CAST 2016 lighting talk and I extended it more at Testival 2016 open session slot where I get excellent feedback from the audience. The goal is to write material that will try to explain this idea in constructive, thoutfll way with experiments feedback that will suport or fail it.

First step, lets detect the symptoms, or reason why you are using UI regression testing.

You have a test plan that only contains a huge list of test cases. Every test case has useful name, and not so useful steps description. Context is a document with a hundred of pages. You are proud, because you wrote that document, and you are the only person that read it.

You have a automated test suite of selenium web driver tests, that have more that a hundred automated test cases, and in order to run it, you need several hours and several Jenkins instances. You have a number of failed tests, and those fails are false positives (test failed but this does not indicate problem in your product).

If this is the case, that means that developers DO NOT WRITE TESTS in their development framework (e.g. Angular, Ruby on Rails, Django,…). Note here that I do not mention unit tests, but DEVELOPMENT FRAMEWORK TESTS (DEFRATE).

There are many DEFRATE broken developers. They know only part of their framework, an usually they are not interested in testing part.

And because of that, you are developers safety net, and you are not doing software testing.

Because software testing is:

screen-shot-2016-10-08-at-5-45-50-pm

Every physician has first to determine the symptoms,  in order to determine disease. I defined the symptoms for  disease called UI regression testing.

Next week I will talk about DEVELOPMENT FRAMEWORK TESTS (DEFRATE).

 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather