Soak testing is
running a system at high levels of load for prolonged periods of time. A soak
test would normally execute several times more transactions in an entire day
(or night) than would be expected in a busy day, to identify any performance
problems that appear after a large number of transactions have been executed.
Also, it is possible that a system may ‘stop’ working after a certain number of
transactions have been processed due to memory leaks or other defects. Soak
tests provide an opportunity to identify such defects, whereas load tests and
stress tests may not find such problems due to their relatively short duration.
The above diagram shows activity for a certain type of site. Each login results
in an average session of 12 minutes duration with and average eight business
transactions per session.
A soak test would run for as long as possible, given the limitations of the
testing situation. For example, weekends are often an opportune time for a soak
test. Soak testing for this application would be at a level of 550 logins per
hour, using typical activity for each login.
The average number of logins per day in this example is 4,384 per day, but it
would only take 8 hours at 550 per hour to run an entire days activity through
the system.
By Starting a 60 hour soak test on Friday evening at 6 pm (to finish at 6am
Monday morning), 33,000 logins would be put through the system, representing 7½
days of activity. Only with such a test, will it be possible to observe any
degradation of performance under controlled conditions.
Some typical problems identified during soak tests are listed below:
Serious memory leaks that would eventually result in a memory
crisis, Failure to close connections between tiers of a multi-tiered
system under some circumstances which could stall some or all modules of the
system.
Failure to close database cursors under some conditions, which would eventually
result in the entire system stalling. Gradual degradation of response time
of some functions as internal data-structures become less efficient during a
long test.
Apart from monitoring response time, it is also important to measure CPU usage
and available memory. If a server process needs to be available for the
application to operate, it is often worthwhile to record its memory usage at
the start and end of a soak test. It is also important to monitor internal
memory usages of facilities such as Java Virtual Machines, if applicable.
Long Session Soak Testing
When an application is used for long periods of time each day, the above
approach should be modified, because the soak test driver is not Logins and
transactions per day, but transactions per active user for each user each day.
This type of situation occurs in internal systems, such as ERP and CRM systems,
where user’s login and stay logged in for many hours, executing a number of
business transactions during that time. A soak test for such a system should
emulate multiple days of activity in a compacted time-frame rather than just
pump multiple days worth of transactions through the system.
Long session soak tests should run with realistic user concurrency, but the
focus should be on the number of transactions processed. VUGen scripts used in
long session soak testing may need to be more sophisticated than short session
scripts, as they must be capable of running a long series of business
transactions over a prolonged period of time.
Test Duration
The duration of most soak tests is often determined by the available time in
the test lab. There are many applications, however, that require extremely long
soak tests. Any application that must run, uninterrupted for extended periods
of time, may need a soak test to cover all of the activity for a period of time
that is agreed to by the stakeholders, such as a month. Most systems have a
regular maintenance window, and the time between such windows is usually a key
driver for determining the scope of a soak test.
A classic example of a system that requires extensive soak testing is an air
traffic control system. A soak test for such a system may have a multi-week or
even multi-month duration.
