Web Service Load Testing:
SoapUI provides extensive load testing functionality allowing you to do the following:
· Functional LoadTesting : validate functionality under load using standard TestCase methods
· Behavioral LoadTesting : analyze performance behaviour under varying load with different load strategies
· Performance LoadTesting : find maximal performance available using thread strategies and Command Line LoadTest execution
· Requirements Driven LoadTesting : define performance requirements and continuously validate using Load Test assertions
Any number of LoadTests can be created for a TestCase (using the TestCases "New LoadTest" popup-menu action), each with different strategies, assertions, etc. to validate/assess a TestCases and its TestSteps performance under different circumstances.
The documentation for load testing in soapUI has been split into the following documents:
· This document gives a background on the soapUI approach to load testing and an overview of the LoadTest Editor
· Load Test Configuration : describes limits and strategies
· Load Test Execution : describes the execution of LoadTests
· Load Test Assertions : specifies the available LoadTest assertions and how they are used
· Load Test Diagrams : describes the available diagrams during LoadTesting
LoadTest Configuration
The actual execution of a LoadTest is configured in two dimensions:
· The LoadTest Limit controls how long a LoadTest should run
· The LoadTest Strategies control how TestCases are scheduled and threaded (i.e. run in parallel)
The LoadTest Limit is specified on the top right of the LoadTest Editor and consists of two settings:
· Limit : specifies for how many TestCase runs or seconds the LoadTest should run
· Limit Type : specifies how to limit the LoadTest; Seconds will run the LoadTest for the specified time, Total Runs will run the underlying TestCase for the specified number of times in total, and Runs per Thread will run each thread that exact number of times (use with care with strategies that change the number of threads).
Please note that the "Total Runs" Limit Type does not guarantee that each thread will hande equal amounts of TestCase executions; if for example setting the Limit to 1000 and configuring 5 threads, each thread will run approximately 200 requests (depending on execution times...). Use the "Runs per Thread" limit for this instead (introduced in soapUI 3.0).
Setting the Limit to 0 with either Limit Types will run the LoadTest indefinitely. A progress bar to the right of the settings will show the current LoadTest progress.
A LoadTest can in soapUI be run using a number of strategies for how TestCases are executed. All strategies have a thread-count allowing the specification of how many TestCases that should be run in parallel. The current strategy is selected using the available Strategy combo, which will show that strategies settings to its right. Some strategies allow interactive changing of the thread count while executing the LoadTest, others don't.
The following LoadTest Strategies are currently available:
· Simple : TestCase execution with a configurable delay
· Variance : TestCase execution varying the number of threads over time
· Burst : TestCase execution in "bursts"
· Thread : TestCase execution with a fixed thread count modification
· Grid : Defines a custom variation of thread count (soapUI Pro only)
· Script : Lets a groovy script control the number of threads (soapUI Pro only)
· Fixed-Rate : Execute a TestCase at a fixed rate (soapUI Pro only)
Since multiple LoadTests with different strategies can be created one could for example create one LoadTest with a simple "base" load using the Simple strategy and another LoadTest using the Burst strategy to monitor how base performance is affected when running both LoadTests simultaneously within soapUI or from the command-line.
The Simple Strategy has the following settings: 
· Test Delay : sets a delay between each TestCase run
· Random : sets how much of the delay to randomize
Use this for simple load testing. For example setting the delay/random to 1000/0.5 will delay each threads TestCase execution by 500 to 1000 ms. Setting the delay to 0 will ignore the random setting and run without any delays.
If setting a delay be aware that the TPS statistic will give misleading results unless the "Calculate TPS on actual time passed" option is selected in the LoadTest Options dialog
The variance strategy has the following settings: 
· Interval : sets the duration of each variance cycle in seconds
· Variance : sets how much the thread-count should be varied during the cycle
Use this strategy for simulating a linearly varying number of threads over time. The strategy will start at the currently configured thread-count, increase to current + (current*variance), decrease to current - (current*variance) and increase back to the current thread-count.
For example setting the thread-count to 20, duration to 480 and variance to 0.5, the strategy will increase from 20 to 30 threads in the first 2 minutes, decrease to 10 threads of the next 4 minutes and then increase back to the original 20 in the last 2 minutes.
The Burst strategy has the following settings: 
· Burst Delay : sets the delay between each burst
· Burst Duration : sets duration for each burst
Use this strategy to simulate a "burst" load and for example monitor system behaviour during the "recovery" periods
For example setting the limit type to seconds, and limit to 600, the thread-count to 30, burst delay to 50, and burst duration to 10 will pause all 30 threads for 50 seconds and run them for 10 seconds repeatedly under 10 minutes.
The Threads Strategy has the following settings: 
· Start Threads : sets the number of threads to start at
· End Threads : sets the number of threads to end at
Use this strategy to vary the number of threads from a start to end value linearly during the entire LoadTest. This can for example be useful to find the thread-count at which the best TPS is achieved.
For example setting the limit type to seconds, and limit to 600, the start threads to 1, and end threads to 30 will increase the number of threads from 1 to 30 over 10 minutes.
The Grid Strategy has the configuration dialog:
Each row in the grid configures the number of threads after the elapsed time since the previous row. In the above example the number of threads will change to 8 after 10 seconds, then to 12 after another 10 seconds, etc. Both Elapsed Time and ThreadCount are stored relatively to the current ThreadCount and limit, so if these are changed the grid values will be recalculated accordingly.
The Script Strategy has the configuration dialog:
Here you create a script that should return the number of threads the TestCase should use. The script is called at the interval specified in the LoadTest Options dialog. The above example would set a random number of threads between 1 and 11 each time it is called.
loop you may want to have "Request level" checked, which applies the fixed rate to sending requests.
LoadTest Execution
When running a LoadTest soapUI internally creates a complete copy of the underlying TestCase for each thread, allowing each TestCase to maintain its own state and properties. Depending on which limit and strategy has been selected, load testing will proceed as configured until the LoadTest terminates due to one of the following:
· It has reached its configured Load Test limit
· It has been canceled by the user with the "Cancel" button on the LoadTest toolbar
· It has been canceled by a LoadTest Assertion when the maximum number of allowed errors for that assertion has been passed
During execution, the following statistics are periodically collected and displayed in the Statistics Table:
· min - the shortest time the step has taken
· max - the longest time the step has taken
· avg - the average time for the test step
· last - the last time for the test step
· cnt - the number of times the test step has been executed
· bytes - the number of bytes processed by the test step
· bps - the bytes per second processed by the test step
· err - the number of assertion errors for the test step
Collection and calculation of this data is performed asynchronously (i.e. independently from the actual TestCase executions) in soapUI, so it should not affect the actual load testing performance.
The Total row shows the statistics for the TestCase itself.
Diagrams
soapUI provides 2 diagrams for behavioral analysis of LoadTest Statistics over time (neither provides any exact values/legends). These are:
· A Statistics diagram showing all statistics for a selected TestStep
· A Statistic History diagram showing a selected Statistic for all TestSteps
Both diagrams share a "Resolution" setting which controls how often the diagram should be updated. This can be set either to "Data" or a number of milliseconds:
· Data - means that the diagram will be updated whenever the main LoadTest Statistics are updated, which in turn is controlled by the "Sample Interval" setting in the LoadTest Options dialog. This is the default setting and sufficient if you have a high throughput of TestCase runs (which will constantly update the Statistics. If you have a LoadTest that has long-running TestCases, this option will result in a diagram that updates very infrequently which may both be confusing and not what is desired.
· XXX - an interval in milliseconds, the diagram will be updated every XXX ms regardless of if the underlying Statistics have changed or not.
Be aware of the fact that all statistics for these diagrams are always collected internally, i.e. you can open the diagram after running your LoadTests and see the results. This is recommended as it does not take any performance from the UI during LoadTesting.
The statistics diagram shows a number of statistics for a selected TestStep or the TestCase Total. The toolbar at the top contains a combo-box for selecting which step to display and a button to the far right for exporting the currently selected steps data to a comma separated file.
The statistics history diagram shows a selected statistic for all TestSteps in the TestCase. The toolbar at the top contains a combo-box for selecting which statistic to display and a button to the far right for exporting the currently selected statistics data to a comma separated file.
Functional Testing of Web Services
You create a test from a WSDL by right-clicking the request and choosing "Add to TestCase". Once you you have this in a TestCase the next natural step is doing functional test of it.
soapUI supports functional testing of Web Services by providing a TestCase metaphor where a number of TestSteps can be executed in sequence. There are currently eleven types of TestSteps available providing for rich testing possibilities. TestCases are further organized into TestSuites of which an arbitrary number can be created within each project.
Functional Web Service testing in soapUI can be used for a variety of purposes:
· Unit testing : validate that each Web Service operation functions as stated
· Compliance testing : validate that the Web Service returns results compliant with its definition
· Process testing : validate that a sequence of web service invocations fulfill a required business process
· Data Driven testing : validate that any of the above works as required with data input from external sources (for example a database or another web service).
Web Service TestSuites
Generating TestSuites
TestSuite Runner
Generating Reports
TestSuite Coverage
soapUI TestCases
The TestCase Editor
No comments:
Post a Comment