TL;DR
In this post I will provide an example how to set the scope of system regression test in order to achieve coverage of features that need to be tested.
Context
The trigger for regression system test was HTTPS only access to client Ruby on Rails application. Prior to that feature, it was possible to use both HTTP and HTTPS protocols. Developers use TDD concept, and prior to my test, all developer tests passed on CircleCI environment.
There was also set of selenium-webdriver tests, but those tests do not cover all application features.
My strategy was to include them in the regression test. All test passed. But I had not finished regression test yet.
I did not know all application features at that time. So I started risk analysis, which features could fail if HTTPS only protocol was introduced to web application. Let’s call OWASP for help.
There is transport layer cheat sheet. Reading through the rules, I pinpointed rules that were potential risk for application functionality:
- Do Not Mix TLS and Non-TLS Content – because browser (modern browsers) will AUTOMATICALLY prohibit access to non HTTPS urls.
- Use a Certificate That Supports Required Domain Names – if this is not the case for your application, browser will present to user a security error
First risk could be mitigated by using the application in Chrome and observing javascript console for mixed content errors. Could I automate that task? First thought would be: write selenium-webdriver test suite that covers all the features! But I do not have that time. Was there a simpler way?
My heuristic was to search all the code base for keywords HTTP and IFRAME.
grep -H -r ‘iframe’ * | grep http:// | grep -v elements.txt | grep -v ‘README.md’ | less
That piped command searched through the all code base in terminal and returned code that uses mixing content.
And we discovered additional issue, it was not possible to immediately set all HTTP urls to HTTPS protocol. Those urls were referencing external applications, like blog. For example, in order to set this blog to HTTPS protocol, I need to buy another plan that costs more money. And I need to have a certificate for tentamen.eu domain. Which brings us to second risk.
Use a certificate that supports required domain names. And this is environment dependant test. This risk was mitigated on my testing environment, but I should also check it on production environment (production is hosted on different domain).
Doing risk analysis is fun. You will learn something new and the most importantly, you will properly set scope for your regression test.
