JMeter: A Data-Driven Test Script - Part 1

In this post and the next I'm going to break down a JMeter script that illustrates a simple data-driven test technique and some of JMeter's components.  The first part will focus on the structure of the script, covering some key logic and configuration controllers; since they warrant a little more attention, the second part will look at the script's request samplers and closely associated components. The script (JMeter script files have a JMX extension) and csv data file used by the script are available for download here in a single zip file.  After unzipping the files, open the JMX file in JMeter (File - Open in the file menu).  By default, the csv file is expected to be in a folder called C:\Temp\PerTableData, but you can put it anywhere you'd like as long as you modify the CSV Data Controller accordingly.

Here's what the script tree should look like once you've opened it:


To see the script in action, click on the node for the View Results Tree listener (where it's easiest to see the script's progress), then click on the Play button at the top of JMeter's main window (the button with the large green arrow).  The script runs against a web service that provides data about the periodic table of elements.  There are four methods provided by the service, which are split into two groups in the script-- three of the methods are element specific (they return data based on an individual element provided as a parameter) and the last simply returns a list of all the elements in the periodic table (if you've read my post on data driven testing in soapUI, this should all look very familiar-- this script runs against the same service and is similarly structured).

Since this is strictly a functional test script, we only need a single user configured in the Test User element at the top of the script:


The HTTP Request Defaults config element at the top of the script specifies a default server, path, etc., to use for request samplers.  Since it's placed at the top of the tree, it's applied to all requests; however, you can override these default values by specifying different values locally (within an individual request's own configuration dialog).  One potential "gotcha" to note: the server name field in this element does not take a full URL-- the "http://" prefix should be omitted when specifying the server.


The "Duration LT 5 Sec" Duration Assertion asserts that all requests should complete within 5000 milliseconds (5 seconds).  As with the HTTP defaults element, its placement at the topmost level of the script tree means it's applied to all requests.


The "ElementRequests Loop" Loop Controller establishes a "branch" to the tree that loops indefinitely (an If Controller placed within the loop will terminate it based on our source data):


Within our loop, the first element is a CSV Data Set Config element:


The element specifies the location of our source CSV file to use to read in variable values-- if you put this file somewhere other than the default location, you'll have to change the Filename entry here.  It also specifies the variable names we're going to use in our script-- each variable name corresponds to a column in our CSV file-- so the first item in each row is assigned to the ElName variable, the second item is assigned to the ElNum variable, etc.  Note that we don't have to do any additional work to manage reading this file-- no maintaining a separate counter to keep our place, for example-- each time it's encountered in our loop, JMeter reads in the next line.  Although we're not using it in this case (we only have one user configured), note that you can "share" data from the CSV file between multiple threads/virtual users via the "Sharing mode" options-- meaning each time any thread encounters the CSV file in its execution, it reads the next line.  So you could have one thread read in data from row 1, the next thread read in data from row 2, etc.

To prevent the loop from running indefinitely, our test requests are nested beneath an If Controller that checks to see if the end of our CSV file has been reached.


When JMeter attempts to read in data from our CSV file after it's already read in the last row, "<EOF>" is returned.  That serves as a flag that it's time to exit the loop-- or more specifically, when "<EOF>" is not returned as the value of the ElName variable, that serves as a flag that it's okay to continue to execute our requests.  The condition statement illustrates how to reference variables: enclose the name in braces and precede it with "$" (remember that our variable names were configured in the CSV Config element).  Note that in our condition statement ("${ElName}"!="<EOF>") the name of our string variable is enclosed in quotes-- this syntax may seem unusual to some; see JMeter's component reference for some more condition statement examples.

Skipping the request samplers and related config elements for now (again, we'll look at those in the next post), the script contains two listener components which collect and report run data.  The Results Tree listener shows request and response data in detail-- even if you don't intend to use this listener in your finalized script, it's good for troubleshooting when you're building a script, allowing you to see exactly what's sent and received and create your assertions accordingly.


In the next post we'll look specifically at the script's request samplers and compare the deprecated SOAP Request sampler with the HTTP Request sampler.

JMeter: An Overview

Apache JMeter is an open source testing tool that's primarily designed for load/performance testing (a key feature is the ability to simulate multiple concurrent users), but it's useful for functional testing as well.  Although I'm going to look at it primarily in the capacity of a web services testing tool a la soapUI, it's capabilities don't end there-- according to its web site, testable resources include "files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more."


The main JMeter window
JMeter's UI features a tree view (on the left in the screenshot above) where you can easily build and visualize your script.  Right-click on a given node in the tree to add child elements or perform other actions (these vary by component type).  You can also drag and drop nodes (including their child nodes) to quickly re-structure your script, moving them before or after other elements or making them children of other nodes.

JMeter's many many script components are divided into several categories.  Check out the extensive component reference for details on all of them.  Some of the key ones:

Threads (Users): Thread elements define and configure the virtual user(s) making requests in your JMeter script.  At least one thread is required before samplers can be added.

Samplers: Test requests are configured using sampler elements, which include samplers for HTTP requests, FTP requests, SMTP requests, and SOAP requests (note, however, that the SOAP request sampler has been deprecated-- the HTTP Request element can be used for SOAP requests as well).

Logic Controllers: Logic controllers add loops and conditional execution of test requests, making it possible to create some fairly sophisticated scripts in terms of flow control.  These include a Loop Controller, ForEach Controller, If Controller, While Controller, etc.

Config Elements: Config elements are used for a variety of configuration-related tasks, including setting/managing variables and creating default parameters to use in request samplers.  The "scope" of some config elements is determined by their level in the script tree.  For example, settings in an HTTP Request Defaults config element are applied to applicable samplers at or below its tree level, allowing you to use a single HTTP Request Defaults element to define the endpoint and path for multiple HTTP Request samplers (saving you the trouble of having to re-enter the same information for each sampler).

Assertions: JMeter Assertions are used to verify expected results, including assertions for expected response time, content, size, etc.  As with config elements, assertions can be applied to multiple samplers by placing them at the same tree level (or higher) in a script.  In the screenshot above, for example, the "Duration LT 5 Sec" assertion is applied to all the samplers in the script since they all fall beneath it in the tree hierarchy.

Listeners: You can add listeners to your script to record, consolidate, and display results in a variety of ways-- listeners include a table view of results, a tree view of results, assertion results summaries, and graphs.

In my next post I'll look at a sample script that should help illustrate some of these components.

soapUI: Scripting Objects - Working with Test Steps

This is a continuation of a series of posts attempting to review some of the objects (and methods) you're likely to encounter when you start scripting in soapUI.  To see the full soapUI javadoc, click here.

Common Test Step Methods

SoapUI test cases can contain a variety of test steps; accordingly, there are a number of test step classes you'll encounter in scripting.  There are no variables like the testSuite or testCase variables available for test step objects in scripts; you can use WsdlTestCase methods like getTestStepList() or getTestStepByName() to access them instead.  This post covers some key methods (primarily dealing with properties) that are available for all test step classes.  Note that the properties we're dealing with here include "built-in" properties-- i.e., properties that aren't created and named by the user (like test case or test suite properties used in property expansions) but properties that are specific to different test step types.  Here are some of the methods common to all test step classes:

- getName() : returns the name of the test step as a String
- getTestCase() : returns the test step's test case as a WsdlTestCase object
- getPropertyList() : returns a Java List of TestProperty objects corresponding to the test step's properties
- getPropertyAt(int index) : given an integer representing the zero-based index of a property in the test step's list of properties, returns a TestProperty object corresponding to that property
- getProperty(String name) : returns a TestProperty object corresponding to the property with name name
- getPropertyValue(String name) : returns the value of the property with name name
- setPropertyValue(String name, String value) : sets the value for the property with name name to value value

Here's a sample script that displays information in the script log for a SOAP Request test step:

firstTestStep = testCase.getTestStepAt(0)
log.info(firstTestStep.getName() + ":")
log.info("     Class = " + firstTestStep.getClass().toString())
log.info("     Parent Test Case = " + firstTestStep.getTestCase().getName())
log.info("Test Properties:")
stepPropList = firstTestStep.getPropertyList()
for(i in stepPropList){
 log.info("     " + i.getName() + " : " + i.getValue())
}

The script log output:



Note the listed test properties (Endpoint, Username, Password, etc.) are all intrinsic properties for SOAP request test steps-- none of these were created by the user.

Test Step Results

You can access test step results via the WsdlTestCaseRunner class's getResults() method.  There are different step result classes corresponding to different test step types; fortunately, a lot of key results information is available through methods common to all of them:

- getTestStep() : returns the test step (as a TestStep object) corresponding to this result

- getTimeStamp() : returns the time the test step was run (as a long-- see the example below for how to convert from a long to a date/time)

- getTimeTaken() : returns the time taken in milliseconds (as a long) to run the test step

- getStatus() : returns the run status of the test step (as defined by the TestStepResult.TestStepStatus enum type; possible values for status include CANCELED, OK, FAILED, and UNKNOWN)

- getSize() : returns the size of response content in bytes (as a long); for non-request test steps (property transfer steps, for example) this is 0

- getMessages() : returns messages generated by the test step

Here's the tree for a sample test suite:

Here's a sample TearDown script that gathers step results and logs info:

//Get Results
tsResults = testRunner.getResults()
for(i in tsResults){
 log.info(i.getTestStep().getName() + " :")
 log.info("   Class: " + i.getClass())
 tStamp = new Date(i.getTimeStamp())
 log.info("   TimeStamp: " + tStamp)
 log.info("   TimeTaken: " + i.getTimeTaken())
 log.info("   Size: " + i.getSize())
 log.info("   Status: " + i.getStatus())
 log.info("   Messages: " + i.getMessages())
}

The resulting output (only the beginning of the output is shown here to save space and avoid repetition):



Note that the step results for different step types represent different classes, but these methods work for all of them.  The PropertyTransfer1 test step also generates its own message, confirming the successful completion of the transfer.