Tips to define the Performance Test Strategy

Best practices in defining performance test strategy

Most of the application would need Load tests, Stress tests and Stability(Endurance) tests to be planned before moving it to production. For data voluminous applications, there might be more emphasis on Volume testing and some applications like Auction site might emphasis on conducting spike tests with sudden spikes. For all other applications, its common that atleast 3 cycles of load tests followed by stress test and endurance test run (atleast for 12 hours) needs to be planned. (This is on a general case, but based on the application context, definetly there will be changes done to this approach).

For any application, irrepective of time availability , its always recommended to start with Baseline test. Baseline test is nothing but one user test which is run to identify the correctness of the test scripts and also it helps is checking whether the application meets the SLAs (Service Level Aggrements) for 1 user load. This values can be used a benchmark for newer versions to compare ths performance improvement.

Next comes the Benchmark test for atleast 15-20% of the target load. It helps in identifying the correctness of the test scripts and tests the readiness of the system before running target load tests.

Its a very good practice to always start with Baseline & Benchmrk tests on any application before conducting the scheduled load tests. I would say, infact its scott barber's way ofcourse.

When it comes to the scheduled tests - Load Tests, always plan to run atleast 3 roundsof load test. Irrespective of doing a slow ramp , its advised to have 3 individual load tests for 50%, 75% and 100% target load. (The load level should be defined based on the system performance rather than just 50%, 75% & 100% target load levels). 

Test Scenario - Have a slow ramp up followed by a stable period for atleast an hour and ramp down. During this stable period, the target user load needs to perform various operations on the system with realistic think times. All the metrics measured should correspond only to the stable period and not during ramp up/ramp down period. Dont conclude any transaction response time just based on 1 or 2 iterations. The server should be monitored atleast for a minimum of 5 iterations (at the same load level), before concluding the response time metrics, because there could be some reason for high /less response time at any single point of time. Thats the reason watch the server performance for atleast 5 iterations at the same load level(during stable load period) and use the 90th percentile response time to report the response time metrics.

Test Metrics - Look for 90th percentile response time and standard deviation, ofcourse along with hits/sec and other server resource monitors. More the deviation(more than 1), more bursty is the graph and hence its recommended to rereun the test. 

Load Tests should be always followed by Stress Tests - Based on the Load test results, slowly increase the server load step by step and find out the server break point. For this test, realistic think time settings and cache settings are not required as the objective of this test to know the server breakpoint and how it fails.

Endurance (Stability) Tests needs to be run atleast for 10-12 hours in order to identify the memory bottlenecks. This need not be run for peak load, but it can be run for average load levels.