Twitter tester ping

After first Zagreb STC meeting, I mentioned that if anyone has any question about testing, I am willing to help via my Skype account (karlo.smid). I picked up the idea from Michael Bolton. One month after the meeting, I held my first Skype session. Here is chat transcript:


Wednesday, December 14, 2011



[8:55:08 PM CEST] Karlo Smid: Hi!
[8:55:12 PM CEST] Hrvoje: Hi!
[8:55:23 PM CEST] Karlo Smid: We can start.
[8:55:32 PM CEST] Hrvoje: Ok
[8:55:38 PM CEST] Karlo Smid: Describe me application technology.
[8:56:01 PM CEST] Hrvoje: Java application running on linux server.
[8:56:04 PM CEST] Hrvoje: With mysql database.
[8:56:40 PM CEST] Hrvoje: Is that what you meant by technology?
[8:56:43 PM CEST] Hrvoje: Could we switch to call?
[8:56:57 PM CEST] Karlo Smid: Ok, so basic application architecture is application server + database.
[8:57:58 PM CEST] Karlo Smid: I would like to continue with skype chat, so I will have archive about this session.
[8:58:05 PM CEST] Hrvoje: Ok.
[8:58:37 PM CEST] Karlo Smid: Do you know approximate number of application users?
[8:59:16 PM CEST] Hrvoje: I am trying to reconstruct testing network, this application will be used for performance and functional testing.
[8:59:54 PM CEST] Hrvoje: In production environment, application has few tens of requests per second. This is pick traffic.
[9:00:16 PM CEST] Hrvoje: Average is 2-3 requests/sec.
[9:00:32 PM CEST] Karlo Smid: Web service requests?
[9:01:16 PM CEST] Hrvoje: No, the most frequent protocols are: HTTP, SOAP, UCP, SMPP.
[9:01:26 PM CEST] Karlo Smid: For test network you mean testing environment?
[9:01:45 PM CEST] Hrvoje: For our customer our application is blackbox, there is no any interface.
[9:01:49 PM CEST] Hrvoje: Yes.
[9:02:39 PM CEST] Hrvoje: What’s bothering me is that they are trying to introduce virtual machines instead of real hardware. I am not sure how application will operate on virtual machines. I do not know how to justify order for real hardware servers.
[9:03:24 PM CEST] Karlo Smid: You finished development of the application?
[9:03:32 PM CEST] Hrvoje: Yes.
[9:03:58 PM CEST] Hrvoje: We used virtual servers for functional tests, are there was no any problem with that.
[9:04:00 PM CEST] Karlo Smid: Do you have any testing server for your application?
[9:04:05 PM CEST] Hrvoje: Yes.
[9:04:21 PM CEST] Hrvoje: 20 for production and 2 to 3 for testing.
[9:05:05 PM CEST] Hrvoje: Application is running in production environment.
[9:05:15 PM CEST] Karlo Smid: How complicate is to simulate application data traffic? I assume that you have done that for one user, because you finished functional testing.
[9:05:49 PM CEST] Karlo Smid: Ok, that mean that you can record live application traffic.
[9:05:55 PM CEST] Hrvoje: Yes.
[9:06:42 PM CEST] Karlo Smid: Which application server do you use in production?
[9:07:27 PM CEST] Hrvoje: tomcat
[9:08:33 PM CEST] Karlo Smid: Lets define your problem. You have in production real linux hardware, and customer wants to replace them with virtual servers. You do not will virtual servers be able to handle real data traffic? 
[9:09:06 PM CEST] Hrvoje: This is not the problem, but it is very close to my problem.
[9:09:23 PM CEST] Karlo Smid: What is your problem?
[9:11:16 PM CEST] Hrvoje: We want to create new test environment for our core application and 5-10 supporting applications. Management approved virtual technology for this test environment.
[9:12:35 PM CEST] Karlo Smid: Have you ever simulated live traffic with some load test tool?
[9:12:43 PM CEST] Hrvoje: I would like real hardware for that new testing environment, but I do not how to argument that to my management.
[9:13:11 PM CEST] Hrvoje: We used JMeter and internally developed applications.
[9:15:37 PM CEST] Karlo Smid: I am using virtual servers (Solaris 10 zones), mostly for functional testing. Zones do not have any restrictions (quotes) on shared hardware, so we are doing load test on them.
[9:16:33 PM CEST] Karlo Smid: For application functionality, virtual environment and OS on the actual hardware are the same.
[9:17:26 PM CEST] Karlo Smid: Of course, OS patch levels, java virtual machine version and settings, OS settings must be the same.
[9:18:14 PM CEST] Karlo Smid: You just have to dimension host hardware for the needed number of virtual machines.
[9:19:09 PM CEST] Karlo Smid: What is virtualization technology?
[9:19:19 PM CEST] Hrvoje: It is very hard to simulate real traffic, because there is great number of traffic use case possibilities.
[9:19:36 PM CEST] Hrvoje: I think that it is VMware.
[9:21:10 PM CEST] Karlo Smid: For virtualization server host machine, you also need to have enough disk space, not just big amount of physical memory. The reason is swap memory configuration on disk drive.
[9:22:05 PM CEST] Karlo Smid: Real traffic consists of great number of different requests?
[9:23:20 PM CEST] Hrvoje: Yes, every mobile operator for every country have different settings for SMS/MMS services.
[9:23:45 PM CEST] Hrvoje: Requests differ in application and database routing.
[9:24:54 PM CEST] Hrvoje: Requests have common parameters, but request handling depends on service type.
[9:25:35 PM CEST] Karlo Smid: Have you heard of  http://www.pairwise.org/?
[9:25:43 PM CEST] Hrvoje: No
[9:27:01 PM CEST] Karlo Smid: This is useful technique when you have great number of test cases. It reduces the number of test cases by keeping test case coverage.
[9:27:36 PM CEST] Karlo Smid: There is tool for that http://www.satisfice.com/tools.shtml
[9:29:24 PM CEST] Karlo Smid: Your goal is to predict the application behavior for those request combinations?
[9:30:22 PM CEST] Hrvoje: Yes, but for load test, not for functional test.
[9:31:02 PM CEST] Hrvoje: Our system has two user roles: operator and customer.
[9:31:28 PM CEST] Karlo Smid: Could you record production application SQL queries?
[9:32:01 PM CEST] Hrvoje: Yes
[9:32:32 PM CEST] Hrvoje: From log files or from source code, but there is a lot of them.
[9:32:49 PM CEST] Hrvoje: And I do not see the purpose of using grep on log files.
[9:32:59 PM CEST] Hrvoje: I think that grep would be hard to do.
[9:33:16 PM CEST] Karlo Smid: Your application has already been deployed in production. What do you want to measure using load test on new testing environment?
[9:34:01 PM CEST] Hrvoje:  For example, we have a new customer and that new customer wants to have 70 SMS/sec. in our application.
[9:34:25 PM CEST] Hrvoje: I do not know is our application  able to handle that peak traffic. And how long it will be able to handle it.
[9:35:26 PM CEST] Karlo Smid: Ok, here is what I would do.
[9:35:53 PM CEST] Hrvoje: I need arguments for real testing hardware, not virtualization solution. Or arguments for the virtualization solution.
[9:36:41 PM CEST] Karlo Smid: 1. Talk with your developers. Do they know which indexes are needed for their SQL queries?
[9:37:23 PM CEST] Karlo Smid: 2. Database has to be BIG, production data scale. What is the number of records in your production system?
[9:37:35 PM CEST] Hrvoje: Yes.
[9:38:13 PM CEST] Karlo Smid: Could you replicate production database in your test environment? Do you have that storage capacity?
[9:38:55 PM CEST] Karlo Smid: What is the deployment of the Java Enterprise application? war or ear archive?
[9:39:38 PM CEST] Hrvoje: I can replicate production database.
[9:39:52 PM CEST] Hrvoje: Several jar archives.
[9:41:08 PM CEST] Karlo Smid: Have you ever used jconsole (it is part of standard JDK)? This is java jmx client for monitoring java virtual machine (jvm) parameters.
[9:41:45 PM CEST] Hrvoje: No, is this similar to htop?
[9:41:56 PM CEST] Hrvoje: Or munin?
[9:43:42 PM CEST] Karlo Smid: No, it monitors jvm, in your case Tomcat instance. jmx port should be activated on Tomcat instance.
[9:45:16 PM CEST] Karlo Smid: Have you ever tuned heap jvm parameters? Garbage collector parameters? These are all important jvm parameters regarding the performance. If you have strong hardware and default settings of those jvm parameters, your hardware is not used in full potential.
[9:46:05 PM CEST] Hrvoje: Yes.
[9:46:35 PM CEST] Hrvoje: System architect and developers are taking care of those parameters.
[9:46:41 PM CEST] Karlo Smid: Using jconsole you will be able to determine are those parameters properly set.
[9:47:02 PM CEST] Hrvoje: What is the format of results?
[9:47:50 PM CEST] Karlo Smid: Regarding the traffic simulation, I would write client for one case of user traffic. I prefer Grinder.
[9:49:40 PM CEST] Karlo Smid: Than I would increase the client traffic in steps of 25 concurrent clients. For that you need also hardware (linux server with quad core and 16GB or RAM could easily produce 70 requests per second.)
[9:50:25 PM CEST] Karlo Smid: Using jconsole I would monitor server jvm. grinder gives out of the box system response times.
[9:52:20 PM CEST] Karlo Smid: Have you ever got java out of memory exception in production system?
[9:52:31 PM CEST] Hrvoje: Sometimes.
[9:54:37 PM CEST] Karlo Smid: This is indication that you have java memory leak in your application, or you should tweak heap memory settings. http://www.ibm.com/developerworks/library/j-leaks/
[9:55:33 PM CEST] Karlo Smid: Have I helped you so for?
[9:55:45 PM CEST] Hrvoje: What is your opinion about jmeter??
[9:56:08 PM CEST] Hrvoje: Yes, you gave me enough input for investigation/thinking.
[9:57:46 PM CEST] Karlo Smid: I am not fond of tools which offer GUI programming because they are not flexible enough. Grinder is Java Load Programming framework.  http://grinder.sourceforge.net/. You should try grinder, you or your developers. I can help to to overcome starting learning curve.
[9:59:05 PM CEST] Karlo Smid: Why grinder? Because it gives you API for concurrent programming. You do not have to worry about deadlocks, concurrent file logging. You can easily use any java API. You can use its out of the box httprequest API class.
[10:00:07 PM CEST] Hrvoje: Than you for your time and will.
[10:01:19 PM CEST] Karlo Smid: See you on the next meeting, I will give grinder presentation.


So we talked about virtualization and performance test. From the chat transcript you can see the difference between load, peak and duration performance test.  We mentioned open source load testing tools Jmeter and grinder. I explained why is Grinder better tool. I also gave load test plan for the application depending on its context.