JMeter: An Overview

Apache JMeter is an open source testing tool that's primarily designed for load/performance testing (a key feature is the ability to simulate multiple concurrent users), but it's useful for functional testing as well.  Although I'm going to look at it primarily in the capacity of a web services testing tool a la soapUI, it's capabilities don't end there-- according to its web site, testable resources include "files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more."


The main JMeter window
JMeter's UI features a tree view (on the left in the screenshot above) where you can easily build and visualize your script.  Right-click on a given node in the tree to add child elements or perform other actions (these vary by component type).  You can also drag and drop nodes (including their child nodes) to quickly re-structure your script, moving them before or after other elements or making them children of other nodes.

JMeter's many many script components are divided into several categories.  Check out the extensive component reference for details on all of them.  Some of the key ones:

Threads (Users): Thread elements define and configure the virtual user(s) making requests in your JMeter script.  At least one thread is required before samplers can be added.

Samplers: Test requests are configured using sampler elements, which include samplers for HTTP requests, FTP requests, SMTP requests, and SOAP requests (note, however, that the SOAP request sampler has been deprecated-- the HTTP Request element can be used for SOAP requests as well).

Logic Controllers: Logic controllers add loops and conditional execution of test requests, making it possible to create some fairly sophisticated scripts in terms of flow control.  These include a Loop Controller, ForEach Controller, If Controller, While Controller, etc.

Config Elements: Config elements are used for a variety of configuration-related tasks, including setting/managing variables and creating default parameters to use in request samplers.  The "scope" of some config elements is determined by their level in the script tree.  For example, settings in an HTTP Request Defaults config element are applied to applicable samplers at or below its tree level, allowing you to use a single HTTP Request Defaults element to define the endpoint and path for multiple HTTP Request samplers (saving you the trouble of having to re-enter the same information for each sampler).

Assertions: JMeter Assertions are used to verify expected results, including assertions for expected response time, content, size, etc.  As with config elements, assertions can be applied to multiple samplers by placing them at the same tree level (or higher) in a script.  In the screenshot above, for example, the "Duration LT 5 Sec" assertion is applied to all the samplers in the script since they all fall beneath it in the tree hierarchy.

Listeners: You can add listeners to your script to record, consolidate, and display results in a variety of ways-- listeners include a table view of results, a tree view of results, assertion results summaries, and graphs.

In my next post I'll look at a sample script that should help illustrate some of these components.

soapUI: Scripting Objects - Working with Test Steps

This is a continuation of a series of posts attempting to review some of the objects (and methods) you're likely to encounter when you start scripting in soapUI.  To see the full soapUI javadoc, click here.

Common Test Step Methods

SoapUI test cases can contain a variety of test steps; accordingly, there are a number of test step classes you'll encounter in scripting.  There are no variables like the testSuite or testCase variables available for test step objects in scripts; you can use WsdlTestCase methods like getTestStepList() or getTestStepByName() to access them instead.  This post covers some key methods (primarily dealing with properties) that are available for all test step classes.  Note that the properties we're dealing with here include "built-in" properties-- i.e., properties that aren't created and named by the user (like test case or test suite properties used in property expansions) but properties that are specific to different test step types.  Here are some of the methods common to all test step classes:

- getName() : returns the name of the test step as a String
- getTestCase() : returns the test step's test case as a WsdlTestCase object
- getPropertyList() : returns a Java List of TestProperty objects corresponding to the test step's properties
- getPropertyAt(int index) : given an integer representing the zero-based index of a property in the test step's list of properties, returns a TestProperty object corresponding to that property
- getProperty(String name) : returns a TestProperty object corresponding to the property with name name
- getPropertyValue(String name) : returns the value of the property with name name
- setPropertyValue(String name, String value) : sets the value for the property with name name to value value

Here's a sample script that displays information in the script log for a SOAP Request test step:

firstTestStep = testCase.getTestStepAt(0)
log.info(firstTestStep.getName() + ":")
log.info("     Class = " + firstTestStep.getClass().toString())
log.info("     Parent Test Case = " + firstTestStep.getTestCase().getName())
log.info("Test Properties:")
stepPropList = firstTestStep.getPropertyList()
for(i in stepPropList){
 log.info("     " + i.getName() + " : " + i.getValue())
}

The script log output:



Note the listed test properties (Endpoint, Username, Password, etc.) are all intrinsic properties for SOAP request test steps-- none of these were created by the user.

Test Step Results

You can access test step results via the WsdlTestCaseRunner class's getResults() method.  There are different step result classes corresponding to different test step types; fortunately, a lot of key results information is available through methods common to all of them:

- getTestStep() : returns the test step (as a TestStep object) corresponding to this result

- getTimeStamp() : returns the time the test step was run (as a long-- see the example below for how to convert from a long to a date/time)

- getTimeTaken() : returns the time taken in milliseconds (as a long) to run the test step

- getStatus() : returns the run status of the test step (as defined by the TestStepResult.TestStepStatus enum type; possible values for status include CANCELED, OK, FAILED, and UNKNOWN)

- getSize() : returns the size of response content in bytes (as a long); for non-request test steps (property transfer steps, for example) this is 0

- getMessages() : returns messages generated by the test step

Here's the tree for a sample test suite:

Here's a sample TearDown script that gathers step results and logs info:

//Get Results
tsResults = testRunner.getResults()
for(i in tsResults){
 log.info(i.getTestStep().getName() + " :")
 log.info("   Class: " + i.getClass())
 tStamp = new Date(i.getTimeStamp())
 log.info("   TimeStamp: " + tStamp)
 log.info("   TimeTaken: " + i.getTimeTaken())
 log.info("   Size: " + i.getSize())
 log.info("   Status: " + i.getStatus())
 log.info("   Messages: " + i.getMessages())
}

The resulting output (only the beginning of the output is shown here to save space and avoid repetition):



Note that the step results for different step types represent different classes, but these methods work for all of them.  The PropertyTransfer1 test step also generates its own message, confirming the successful completion of the transfer.