TL;DR

The first talk of the second day was fear for every manager, a proposition to delete test scripts that do not bring any value to a project.

This talk Should we just… delete it?! was given by João Proença. I met him at EuroTestConf 2018. I enjoy software testing conversations with people who have, just like me, UTF-8 characters as part of their name. You can catch this talk at the original TestBash that will be held in Brighton.

João started with an explanation of why we are all afraid to delete tests or production codes that do not bring value to the project. We got two questions:

Group A has a 100% chance to get 900 EUR, and Group B has 90% of getting 1000 EUR and 10% of getting 0 EUR. What is your choice?

The majority of the audience chose Group A.

Group A has a 100% chance to pay out 900 EUR, and Group B has a 10% chance to pay out 0 EUR and a 90% chance to pay out 1000 EUR.

The majority of the audience chose Group B. The reason is Loss Aversion:

In cognitive psychology and decision theory, loss aversion refers to people’s tendency to prefer avoiding losses to acquiring equivalent gains: it is better to not lose $5 than to find $5 [Wikipedia].

Our decisions are biased for various reasons.

The IKEA effect is a cognitive bias in which consumers place a disproportionately high value on products they partially created [Wikipedia].

In economics and business decision-making, a sunk cost (also known as retrospective cost) is a cost that has already been incurred and cannot be recovered [Wikipedia].

In psychology and behavioral economics, the endowment effect (also known as divestiture aversion and related to the mere ownership effect in social psychology[1]) is the finding that people are more likely to retain an object they own than acquire that same object when they do not own it [Wikipedia].

Do you still found your manager guilty for not allowing you to delete those test scripts?

João then make a case on how to calculate value vs. cost of existing test scripts, so you could make a business decision based on evidence.

Test script cost could be divided:

• Coverage
• Ownership
• Development
• Maintain

The infrastructure feedback loop is an important metric that defines value. To try to determine maintain cost, we ask the following questions:

• What we check with test scripts?
• Do we have redundancy?
• On what level do we test, unit, integration, e2e?
• Do we have flaky tests?
• How often we update the golden master?
• What is the time spent to fix the test script?

We also need to do a risk analysis. You use the following equation:

Risk = Impact x Probability

It is important to be aware that probability changes over time:

• Feature lifecycle
• Product history
• Tech evolves

There are test scripts alternatives:

• Monitor (Observability)
• Dogfooding (use your own product in beta)

And in the end, we got following the delete traceability matrix:

There was time for Q&A. I asked does he have any experience in automating test/feature traceability. His team is using custom code tags, which they put manually to create traceability. Then, traceability is automated.

When you start a new feature, first start with risk analysis. Risk storming is one example of how to do it.