So you are in position to use some programming language in order to automate test. Welcome to the rabbit hole of software testing! If you would like to know more about software development in software testing, start with this excellent blog from Michael Bolton.
Now that you know difference between checking and testing, I will give an rspec example of proper automated check in REST API test.
In my example, REST API response is in JSON format. In case of successful response, message is:
{response = ‘success’}
and in case of some error, response message is:
{response=’error’, error=’error message’}.
How should we code in rspec response message check when we expect successful response?
First option:
response_message[‘response’].should eq(‘success’)
What is wrong with this check? What if we receive error message? Failure in that case would be:
expected response_message[‘success’] to be ‘success’ but ‘error’ received.
Do you know from test fail message what caused error?
Automated test has to be designed in following manner:
- you are able to determine cause of fail directly from fail message
- test has to be designed so that checks one and only one thing
- test is independent from other tests. In rspec world that means it always produce same result when we use rspec option –order=random
I can't think of any good reason to make checks independent. I keep seeing that advice, but I don't know where it comes from.
It may be that you choose to name things that are independent as different checks. That's your choice. Therefore a check I might write that has 50 assertions in it would be one "check" to you.
However, I can easily think of examples where I conduct an automated tour of a product and place what I would call checks all along that tour. They are not independent, as they are built along one evolving state, and it is not helpful or reasonable to make them independent, because that merely introduces a lot of inefficiency in constantly recreating the same states over and over.
If you are going to give that advice, you ought to show some engineering reason, otherwise it looks like a game where our automation is created not to be poweful and effective, but rather shallow and easy to work with.
The nearest you come to that reasoning, here, is that you imply independence means that you can use random ordering to find interesting state bugs. Well, okay. But you should precede that with a conditional: "If you want to make use of the –random feature of rspec then make your checks independent."
Personally, I don't use rspec. I like to build my own frameworks, thank you. But whenever I do use a tool I don't want that tool to tell me that I have to warp my test strategy to serve the tool. The tool must serve my strategy.
Hi James, thank you for your comment.
I agree with your reasoning that tool must serve tester strategy, not vice versa. Here is more details what do I mean by independent test in context of rspec tool and REST API functionality that I am testing.
–"test has to be designed so that checks one and only one thing" – I was not clear enough with that statement. Here by test I meant rspec "it" statement. One it should have only one assert (check). Test could consist of several it statements that are dependent. This is context rspec statement
By independent test I mean one REST API interaction with the application. Precondition for REST API interaction is particular application state (application is put in that state with other REST API interaction). If tested REST API interaction fails, it puts application in different state. So next test has to call test precondition that puts application in test precondition state. In world of rspec this is before(:each) block. You are right that because of rspec design, rspec creates initial state over and over again. But it should create only for failing tests. rspec is missing block after(:exception).
Could you please provide more details how do you build your own testing framework? Do you start from scratch, or do you extend some existing open source framework?
Regards, Karlo.