Directions for fellow Performance Testers

Before certifying the application performance,

are you satisfied with your own performance? are you sure that you are conducting effective performance test? Have you ever thought how to ensure the your performance test strategy and are you sure that your performance test report is bug free?

The following tips would help you to get more maturity and to do your job better

Yes..You need to be comfortable in the terminologies used in the performance engineering field and need to learn one or two popular and effective performance test tools and develop bottleneck analysis skills. That's the basic skills required for being a performance tester. Then you are matured enough to look back at your performance and look for how effective you are in your activities.

That's where, you need to start looking at Analytical modeling and simulation modeling approaches to move to your next level of expertise.

Foremost thing is that, understand the operational laws and start applying those laws in your day to day test activities.

Secondly start learning about Workload Modeling concepts and Statistical distribution fitting and apply those learning's to define performance testing goals/SLAs.

Thirdly start learning about Demand planning and Performance prediction techniques. Learn Mean value analysis and Erlang's method.

Fourth step is start learning about Capacity Planning.

This is the second level of maturity you need to gain. The next advanced third level is moving towards simulation modeling and advanced capacity planning.

Hope this helps to set your career path and plan to work towards it.

How do you know if your Load Test has a bottleneck

The bottleneck in a system may not be obvious. (Life would be easier but less fun if there where always easy to find). This is because there are two types “hard” and “soft”. Hard bottlenecks are the ones where a resource such as a CPU is working flat out which limits the ability of the system to process more transaction. While a soft bottleneck is some internal limit such at number of threads or connections that once all used limit the ability to process more transaction. Therefore, how do you find know if you have a bottleneck. If you are looking at the results from a single load test you may not know you will need to run multiple load tests at different numbers of virtual users and then see if you number of transactions per second increase with each increase in virtual users. The results can be seen in the two graphs below. The first shows how the throughput (transaction per seconds) increases and levels off when saturated and the second shows the response time. You will probably have heard the express below the knee of the curve and this is an the point that is to the left of the bend in the response time graph.

Throughput Graph

The graphs above where actually generated using a spreadsheet model for the performance of a closed loop model. This is like LoadRunner and other testing tools where the are a fixed number of users that use the system then wait and return to the system. The reality is that the performance graphs may look different from the expected norm. An example is shown below from a LoadRunner test the first graph shows how the number of VUser where increased during the test and the second graph shows the increase in response times. In this case the jump in response time is dramatic. However, in some cases the increase in response time will be less dramatic as the system will start to error at high loads which will distort the response time figures.




Having discovered there is a bottleneck in the system then you have to start looking for it.

Importance of Little's Law

Little's law is quite simple and intuitively appealing.

The law states that the average number of customers in a system (over some time interval), N, is equal to their average arrival rate, X, multiplied by their average time in the system, R.

N = X . R (or) for easy remembrance use L = A . W

This law is very important to check whether the load testing tool is not a bottleneck.
For Example, in a shop , if there are always 2 customers available at the counter queue , wherein the customers are entering the shop at the rate of 4 per second , then time taken for the customers to leave from the shop can be calculated as

N = X. R

R = 2/4 = 0.5 seconds


A Simple example of how to use this law to know how many virtual users licenses are required:

Web system that has peak user load per hour = 2000 users
Expected Response time per transaction = 4 seconds
The peak page hits/sec = 80 hits/sec

For carrying out Performance tests for the above web system, we need to calculate how many number of Virtual user licenses we need to purchase.

N = X . R
N = 80 . 4 = 320

Therefore 320 virtual user licenses are enough to carry out the Load Test.

Welcome to Performance Testing Blog

Hi, I am Sudhakar. Currently am working as a Performance Test Engineer @MNC company Hyderabad.

I am providing training sessions (online & regular classes) on performance testing with load runner and rational performance tester (RPT) tool with real time process.

If anybody interested to learn performance testing tools, you can reach me ”sudhakar6r@gmail.com”.

Siebel Correlation with custom code in LoadRunner

“Siebel _Star_Array” is a function generated by LoadRunner for correlating Siebel field values separated by Token "*". A sample format of the server response is given below and these field values are required to pass to the subsequent requests.

"1*N9*1-18506217*GEOFFTH4*Open10*Unassigned3*OEM12*Hardware-OEM14*Administrative1*01*010*Symptom X18*MSE-Perf7*GEOFFTH1*112*Geoff Thomas1*10*6* Normal19*10/16/2002 03:40:127*Sev - 47*1-13NY5"

Last Character/s (Highlighted in Red) in each of the fields denotes the length of the next field values. Because of which left and right boundaries will be dynamic and difficult to correlate.

The below "siebel _Star_Array” function provided with LoadRunner has some limitations and does not work at all times. It is it is tough to debug the errors without the source code

web_reg_save_param("WCSParam96",
"LB/IC=`ValueArray`",
"RB/IC=`",
"Ord=10",
"Search=Body",
"RelFrameId=1",
"AutoCorrelationFunction=flCorrelationCallbackParseStarArray",
LAST);

The sample code I had created can be used to parse the response and use it for correlation when compared to the Loadrunner Automactic Correlation

vuser_init()
{

char str[1024];
char separators[] = "*";
char *token;
char arrValues[50][20];
char arrFinalValues[50][20];
int i;
int j;
int length_old;
int length_new;
char newlength[2];
char actualValue[20];

/****************** Sample Text format****************************** */

strcpy(str, "1*N9*1-18506217*GEOFFTH4*Open10*Unassigned3*OEM12*Hardware-OEM14*Administrative1*01*010*Symptom X18*MSE-Perf7*GEOFFTH1*112*Geoff Thomas1*10*3 - Normal19*10/16/2002 03:40:127*Sev - 47*1-13NY5");

lr_output_message("%s",str);

/***** The following code will be used to split the text into strings based on the token *******/
token = (char *)strtok(str, separators); /* Get the first token */
if(!token) {
lr_output_message("No tokens found in string!");
return( -1 );
}

i=0;
while( token != NULL ) { /* While valid tokens are returned */
strcpy(arrValues[i],token);
lr_output_message("Initial array values is %s",arrValues[i]);
token = (char *)strtok(NULL, separators); /* Get the next token */
i++;
}

/*******************************************************************/
/*************** To remove Trail charecters ***********************/

for (j=0; j less than i; j++) //use lessthan sysmbol
{
if (j==0) {
strcpy(arrFinalValues[j],arrValues[j]);
length_old=strlen(arrValues[j]);
}
else{
length_new=strlen(arrValues[j]);
strncpy(arrFinalValues[j], arrValues[j], length_old);
if(length_new>length_old+1){
sprintf(newlength,"%c%c",arrValues[j][length_old],arrValues[j][length_old+1]);
length_old=atoi(newlength);
}
else{
sprintf(newlength,"%c",arrValues[j][length_old]);
length_old=atoi(newlength);
}//End of Else
}//End of Out Else
}//End of For

/* Final Data in the Array are */
for (j=0; j less than i; j++)//less than symbol
{
lr_output_message("Values after removing tail charecters %s",arrFinalValues[j]);
}
return 0;
}

How to determine two performance runs are statistically different?

Performance tests are good examples of Normal Distribution. A normal distribution of data means that most of the examples in a set of data are close to the "average," while relatively few examples tend to one extreme or the other.

According to statisticians, Two tests are considered statistically different if it is unlikely to have occurred by chance. we can be 99% sure that averages from two runs are really different only when the other average is more than 2.57 standard deviations away.

The same principle can also be applied to performance tests to determine results obtained are statiscally different.

For example, If the Average response time for transaction A is 1 sec during the first run with the standard deviation of 0.2 and for the second run average response time for the same transaction A is 1.2 sec. The difference between the two transaction response time is 1.2 – 1.0 =0.2, which is 1 standard deviations away (Standard deviation observed during the first run is 0.2) on the average. If the difference is 2.57 standard deviations away(we are 99% sure) then the results are statistically different.
 

Conversely we can even calculate the statistical limits using the formula below
If R1 is average response time and SD is the standard deviation for a performance run then the average response time for the second run should not exceed R1 + (SD* 2.33) or should not be less than R1 - (SD* 2.33). For the above example response time for the second run should not exceed 1 + (0.2* 2.33) =1.466 on the positive side of the average and 1 –(0.2*2.33) = 0.534 on the negative side of the average.

Working with LoadRunner Sybase Ctlib protocol

The goal of this document is to bring together necessary information to help those users who are involved in scripting Client Server Database protocols using LoadRunner.


The document can be used as a basis for scripting, enhancing and debugging any of the following protocols

Ø Sybase CTlib
Ø Sybase DTlib
Ø Informix
Ø MS SQL Server
Ø Oracle 2-Tier
Ø ODBC
Ø DB2 CLI
Ø ERP/CRM Siebel Vuser scripts


This document is based on my experiences using LoadRunner with Sybase Ctlib protocol

Click here to download the document