TL;DR

It was time for a first regular talk with the duration of 30 minutes. Among the three options, I choose talk that tries to resolve the problem of UI site regressions.

Jeremias Rößler – Test Automation without Assertions

Jeremias is on the path to solve the problem of UI Visual regressions issues. He opened his talk with the famous umbrella test, where acceptance criteria are:

When I open my umbrella, the umbrella is opened.

In the video, the umbrella opens, but it fells from the handle so we can not use it. This is also the problem of every Selenium test. Every test can only guarantee what it checks in assertions (expect clause). HTML is a nasty beast, so on even a simple page, we could have a thousand checks. We could check stylesheet properties, like font size, line sizes, positions (absolute and relative), mobile view.

There is also a problem with HTML locators. They could vanish or change value or XPath. In both cases, the traditional Selenium test would fail. Jeremias punch line for his tool recheck (the company is retest), is:

Regression testing is an extension of version control.

In regression testing, we compare saved output to the current one. Two problems that do not make this easy are:

• noise – some parts do change, and this is ok
• redundancy in test code – check same elements many times

Pixel by pixel comparations are not effective because they depend on screen resolution, and we could not compare them in version control. In the Recheck approach, they use a text file, recheck.ignore. Something like .gitignore. In that file, they support rich syntax for various rules which page elements we want to ignore (noise and redundancy). Human input is to determine the scale of how much we ignore. It is on the tester that needs, depending on the context, to hit a sweet spot.

Google has an in-house solution for UI regression where developers with special HTML element Unicorn marked elements that were changed in a pull request. Using that visual information, the tester could spend more time testing these parts of UI. The problem was that those Unicorns could slip into production :). Have you ever seen anything like that?

In this presentation, I heard for the first time term Golden Master, which is a fancy word for something that we trust that is ok (previous version of the page).

Recheck has a UI component (and Chrome extension) that helps you to speed up to compare to the current version and Golden Master. In that tool, you can mark what is desired regression,  and Golden Master is automatically updated. There is also a CLI tool that helps you to quickly update existing test scripts.

You still write Selenium tests, but only to drive the application to the desired state. You then save golden master, no need for assertions. You can give it a try from recheck-web GitHub repo.

Here is a one-minute video:

a