HP Diagnostics and LoadRunner Integration

The HP Diagnostics integration with LoadRunner allows you to monitor and analyze the performance of Java 2 Enterprise Edition (J2EE), .NET-connected, SAP, Oracle, and other complex environments.

Specifying the Diagnostics Server Details: The first time you use LoadRunner to capture J2EE or .NET diagnostics data, you need to identify the machine on which the Diagnostics Server in Commander mode is running, and the port that it is using for communication with LoadRunner. You must update this information if you want to integrate with a different Diagnostics Server in Commander mode, or if you change the port it is using. To update the LoadRunner configuration settings for HP Diagnostics:

1. Select Start > Programs > HP LoadRunner > LoadRunner to open the LoadRunner launcher window.

2. From the LoadRunner launcher window menu, select Configuration > Diagnostics for J2EE/.NET Setup to open the Diagnostics for J2EE/.NET Setup dialog box. Enter the details for the Diagnostics Server in Commander mode.

3. Click Test to verify that you entered the correct information for the Diagnostics Server in Commander Mode and that there is connectivity between the Diagnostics Server in Commander Mode and LoadRunner. Click OK to complete the configuration process.

Configure LoadRunner Scenarios to use HP Diagnostics

1. Before configuring your scenario for Diagnostics, ensure that the application server that you are monitoring has been started.

2. In the Controller, open the relevant load test scenario (File > Open) or create a new scenario (File > New).

3. Select Diagnostics > Configuration to open the Diagnostics Distribution dialog box.

4. Set the percentage of Vusers to participate in the HP Diagnostics (J2EE/.NET Diagnostics) monitoring. The maximum percentage of Vusers for which HP Diagnostics (J2EE/.NET Diagnostics) data can be collected is 100%, unless you have enabled other types of diagnostics. In this case, the percentage of Vuser participation in HP Diagnostics (J2EE/.NET Diagnostics) cannot exceed the maximum of any of the other types of diagnostics that you enabled.

5. In the Online & Offline Diagnostics section of the Diagnostics Distribution dialog box, next to J2EE/.NET Diagnostics, click Configure. The J2EE/.NET Diagnostics Configuration dialog box opens.

6. Select Enable J2EE/.NET Diagnostics

7. In the Select probes list, select the probes to be included in your load test scenario.

8. If the Diagnostics Server (or a Diagnostics Server in Mediator mode in a distributed environment) is located behind a firewall, select there is a firewall between the mediator and the Controller, and enter the name of the MI listener server in the MI listener server box. If there is a firewall between the LoadRunner Controller and the Diagnostics Server involved in a load test, you must configure the Controller and the Diagnostics Server to use the MI Listener to enable the transfer of the offline analysis file.

9. To capture a percentage of server requests which occur outside the context of any Vuser transaction select Monitor server requests.

10. To investigate any issues that you have with the connections between the Diagnostics components, click the Troubleshoot Diagnostics for J2EE/.NET connectivity link. This will open the HP Diagnostics System Health Monitor in a new browser window.

During a LoadRunner load test scenario, you can view HP Diagnostic’s data for the whole scenario or you can drill down to HP Diagnostic’s data from a particular transaction. After you have run your scenario, you can use HP LoadRunner Analysis to analyze offline Diagnostics data generated during the scenario.

Performance bottleneck symptoms

CPU Bottleneck Symptoms:

Symptoms for CPU bottlenecks include the following; The Processor(_Total)\% Processor Time (measures the total utilization of your processor by all running processes) will be high. If the server typically runs at around 70% or 80% processor utilization then this is normally a good sign and means your machine is handling its load effectively and not underutilized. Average processor utilization of around 20% or 30% on the other hand suggests that your machine is underutilized and may be a good candidate for server consolidation using Virtual Server or VMWare.

Further to breakdown this %processor Time, monitor the counters – Processor(_Total)\% Privileged Time and Processor(_Total)\% User Time, which respectively show processor utilization for kernel- and user-mode processes on your machine. If kernel mode utilization is high, your machine is likely underpowered as it’s too busy handling basic OS housekeeping functions to be able to effectively run other applications. And if user mode utilization is high, it may be you have your server running too many specific roles and you should either beef hardware up by adding another processor or migrate an application or role to another box. The System\Processor Queue Length(indication of how many threads are waiting for execution) consistently greater than 2 or more for a single processor CPU is a clear indication of processor bottleneck. Also look at other counters like ASP\Requests Queued or ASP.NET\Requests Queued as well.

Tips to find out Application server bottlenecks:

1. A high increase in application server processing time when the load is increased.
2. One or more page components take more time when the same request db call is   taking less execution time.
3. The Static files are having less response time whereas the dynamic contents (servlets, jsp, etc) take more time.
4. Network delay is negligible.
5. Home Page gets displayed in few seconds even during the stress period (as it is fetched from the web server).
6. Hits/sec & Throughput remains less.
7. If the CPU/ Memory/Disk of the App server has any bottleneck symptoms.
8. If the HTTP / HTTPS connections established doesn’t increase proportionally with the load.
9. If the new connections established is very higher & the reused connections are very less.

Tips to find out Web server bottlenecks:

1. Increased ‘Server Time’ breakup
2. One or more page components of transaction takes more time where in the DB  query is having less execution time.
3. The static files are having high response time than the dynamic contents (servlets, jsp, etc)
4. Network delay is negligible.
5. Home Page takes more time for display.
6. Hits/sec in the web server is very less.
7. If the CPU/ Memory/Disk of the web server has any bottleneck symptoms.

Hardware Malfunctioning Symptoms:

1. System\Context Switches/sec (measures how frequently the processor has to switch from user- to kernel-mode to handle a request from a thread running in user mode). If this counter suddenly starts increasing however, it may be an indicating of a malfunctioning device, especially if you are seeing a similar jump in the Processor(_Total)\Interrupts/sec counter on your machine.
2. You may also want to check Processor(_Total)\% Privileged Time Counter and see if this counter shows a similar unexplained increase, as this may indicate problems with a device driver that is causing an additional hit on kernel mode processor utilization.
3. If Processor(_Total)\Interrupts/sec does not correlate well with System\Context Switches/sec however, your sudden jump in context switches may instead mean that your application is hitting its scalability limit on your particular machine and you may need to scale out your application (for example by clustering) or possibly redesign how it handles user mode requests. In any case, it’s a good idea to monitor System\Context Switches/sec over a period of time to establish a baseline for this counter, and once you’ve done this then create a perfmon alert that will trigger when this counter deviates significantly from its observed mean value.

Memory Bottleneck Symptoms:

When it comes to the System memory, there are 3 things to monitor:

1. Monitor Cache (Hits/Misses),
2. Monitor Memory (Memory Available/sec, Process/Working Set),
3. Monitor Paging (Pages Read/Sec, Pages Input/Sec, Page Faults/Sec, % Disk  Processing)Memory\Available Bytes,

If this counter is greater than 10% of the actual RAM in your machine then you probably have more than enough RAM and don’t need to worry. The Memory\Pages/sec counter indicates the number of paging operations to disk during the measuring interval, and this is the primary counter to watch for indication of possible insufficient RAM to meet your server’s needs. You can monitor Process(instance)\Working Set for each process instance to determine which process is consuming larger and larger amounts of RAM. Process(instance)\Working Set measures the size of the working set for each process, which indicates the number of allocated pages the process can address without generating a page fault. A related counter is Memory\Cache Bytes, which measures the working set for the system i.e. the number of allocated pages kernel threads can address without generating a page fault. Finally, another corroborating indicator of insufficient RAM is Memory\Transition Faults/sec, which measures how often recently trimmed page on the standby list are re-referenced. If this counter slowly starts to rise over time then it could also indicating you’re reaching a point where you no longer have enough RAM for your server to function well.

Disk Bottleneck Symptoms:

A bottleneck from a disk can significantly impact response time for applications running on your system. Physical Disk (instance)\Disk Transfers/sec counter for each physical disk and if it goes above 25 disk I/Os per second then you’ve got poor response time for your disk. By tracking Physical Disk(instance)\% Idle Time, which measures the percent time that your hard disk is idle during the measurement interval, and if you see this counter fall below 20% then you’ve likely got read/write requests queuing up for your disk which is unable to service these requests in a timely fashion. In this case it’s time to upgrade your hardware to use faster disks or scale out your application to better handle the load. Look for the Physical Disk (instance)\Average Disk Queue length & Physical Disk (instance)\Current Disk Queue length parameters to get more details on the queued up requests.

Network Performance/Bottlenecks:

The first step in monitoring is to monitor the network performance, to make sure your network performance is good. There are some simple ways to do so. First monitor whether you are getting the same bandwidth which you are supposed to get. The easiest way to find out is to check the current bandwidth counter with your expected bandwidth. Also verify the rate at which the server sends and receives the data. Network performance depends on 2 factors, network cards and interfaces (Switches/Routers) configured on the servers.

Here are some of the counters to find network bottlenecks:

Network Interface: Current Bandwidth

This counter determines your current bandwidth of the network interface. Capture this counter value and correlate with bytes receives/sec, bytes send/sec and bytes total/sec.

If the bytes total should be at least half of your total bandwidth .If not so then we can confirm a network bottle neck

Network Interface: Bytes Total/sec:

To determine if your network connection is creating a bottleneck, compare the Network Interface: Bytes Total/sec counter to the total bandwidth of your network adapter card. To allow headroom for spikes in traffic, you should usually be using no more than 50 percent of capacity. If this number is very close to the capacity of the connection, and processor and memory use are moderate, then the connection may well be a problem. To determine the network utilization (throughput on a server’s network cards), you can check the following counters:

1. Network\Bytes Received/sec
2. Network\Bytes Sent/sec
3. Network\Bytes Total/sec
4. Network Current Bandwidth

If the total byte per second value is more than 50 percent of the total network utilization under average user/work load, then your server is having some problems under peak load conditions. Make sure you compare network counter values with Physical Disk\% Disk Time and Processor\% Processor Time utilization. If the disk time and processor time values are low but the network values are very high, there might be a problem with your network. There are 2 ways to solve this problem:

1.     By optimizing the network card settings

2.     By adding an additional network card.

Web Page Rendering

Below are the step-by-step descriptions of how page get displayed in your browser when you simply enter some URL in address bar:

1. You type an URL into address bar in your preferred browser.
2. The browser Parser the URL to find the Protocol, host, port and path.
3. Its forms a HTTP request.
4. To reach the host, it first needs to translate the human readable host into an IP number and it does this by doing a DNS lookup on the host.
5. Then a Socket needs to be opened from the user’s computer to that IP number on the port specified (most often port 80).
6. When a connection is open, the HTTP request is sent to the host.
7. The host forwards the request to the Server software (most often Apache) configured to listen on the specified port.
8. The server inspects the request (most often only the path) and launches the Server plug-in needed to handle the request (corresponding to the server language you use PHP, Java, .Net, Python).
9.  The plug-in gets access to the full request and starts to prepare a HTTP response.
10.  To construct the response database is (most likely) accessed. A database search is made, based on Parameters in the path (or data) of the request.
11.  Data from the database, together with other information the plug-in decides to add, is combined into a long string of text (probably HTML).
12. The Plug-in combines that data with some Meta data (in the form of HTTP headers) and sends the HTTP response back to the browser.
13.  The browser receives the response and Parse the HTML (which with 95% probability is broken) in the response.
14.  A DOM tree is built out of the broken HTML.
15.  New requests are made to the server for each new resource that is found in the HTML source (typically images, style sheets and Java files). Go back to step 3 and repeat for each resource.
16.  Style sheets are parsed and the rendering information in each gets attached to the matching node in the Dom tree.
17. Java Script is parsed and executed and DOM nodes are moved and style information is updated accordingly.
18.  The browser renders the page on the screen according to the Dom tree and the style information for each node.
19. You see the page on the screen.

And we get annoyed why the response time is very high but now at least I have some documentation to look at, while waiting the remaining fractions of a second before the page renders.