Sikuli IDE: Using Pattern Objects

If you go back to the script illustrated in the last post and look at the code for click() or wait() calls in IDLE or some other IDE for Python (other than Sikuli's own editor), you'll see something like the following:
...
wait("FileEditForm.png",5)
...
click("1351739445615.png")
Beneath all the source images in our script (as displayed in Sikuli's IDE) are strings matching the image file names.  When rendering the script in its IDE, Sikuli interprets these strings as instances of its Pattern object, replacing them with the images they represent.  It's important to recognize that we are not limited to using Pattern objects within click() and wait() calls directly-- we can manipulate them programmatically to make scripts more flexible and/or efficient.

To illustrate, let's say we have an app that looks like this:


When the user clicks on any of the color buttons along the bottom of the app form, the large "Color" text changes to match that color.  A straightforward way of writing a test script would be to write four pairs of click() and exists() methods, with each click() on one of the four buttons followed by an exists() check on the "Color" text (to verify it's displaying the right color).  But keeping Pattern objects in mind we could also organize our script like this:


A Sikuli script with Pattern objects organized in a list

In this script, the Pattern objects (images) are organized into a list of tuple pairs; the first image in each pair is the button being clicked, and the second is the "Color" text as we expect it to appear when each button is clicked.  We can use the image list with a for loop to check all four buttons. Using a list like this opens up other possibilities, too-- for example, we could easily step through it randomly instead of following a set order. Obviously, this is an oversimplified example, but it still reflects a fundamental idea behind UI testing that can be applied to more complex applications: interacting with one UI element (via a click, for example), then verifying expected behavior by confirming the existence (or non-existence) of some associated visual "cue".

A few words about some "gotchas" when working with source images: going back to the example in the first paragraph about file name strings, there's no path information necessary with this particular example because the files are located in the same directory as the Sikuli script.  Of course, if you wanted to put your source images in a different directory, you'd have to add corresponding path information to the file name string.  Be aware that as of build 905 of Sikuli X, moving the images out of the main .sikuli folder for your script can also result in problems with rendering them properly in Sikuli's IDE-- you'll see some errors appear in the Message pane at the bottom of the IDE when loading your script, and where you'd expect to see images you may see the underlying strings instead. Here's what my image list looks like in Sikuli's IDE after moving the images to a sub-directory in the .sikuli folder called "Imgs" (and modifying the script accordingly):

image_list = [("\\Imgs\\EE.png","\\Imgs\\Color.png"),
        ("\\Imgs\\1354395248414.png","\\Imgs\\C0l0r.png"),
        ("\\Imgs\\1354395299725.png","\\Imgs\\C0l0r-1.png"),
        ("\\Imgs\\1354395332765.png","\\Imgs\\C0l0r-2.png")]

The script still runs successfully, but the images themselves are not loaded and displayed in the IDE.

One alternative way to deal with source images located in directories other than the .sikuli folder is the addImagePath() method.  Sikuli maintains an image search path-- a list of directories that it will search when a given image can't be found in the active .sikuli folder.  So for example, if you moved an image into a directory called "C:\Temp\Images", you could just add the directory to the image search path using the following code:

imgPath = list(getImagePath())
if 'C:\\Temp\\Images' not in imgPath:
    addImagePath('C:\\Temp\\Images')

Subsequently, the full path for any images in that directory would not have to be specified-- you could just specify the image file name by itself; Sikuli would automatically include the "C:\Temp\Images" directory when looking for the image.  Note that while this method does save some time typing out paths when specifying images, it may still result in the same problems explained above when using non-.sikuli folders for images.

Sikuli IDE: A Basic Walkthrough

This is a brief Windows-oriented tutorial that walks through a Sikuli IDE test script (there are tutorials provided on the Project Sikuli site, too).  This script demonstrates how to create and manipulate the Sikuli App object and perform some basic UI actions using the Notepad application.

Here's the starting point for your script:

NPPath = 'C:\\Windows\\notepad.exe'
#Open Notepad using the App object
NPApp = App.open(NPPath)
#Wait until the menu bar is visible, then display message
#that the window is open.
wait("Your Notepad Menu Bar",5)
popup("Window is open!")
menuBar = getLastMatch()#Takes advantage of wait function side effect
#Highlight the menu bar region for 3 seconds then click on text
menuBar.highlight(3)
menuBar.click('Help')
#Define a new region using 'Help' text as starting reference
dropDownRegion = Region(menuBar.getLastMatch().getX(),menuBar.getLastMatch().getY(),300,100)
dropDownRegion.click('About')
wait(3)
click("Your OK Button")
type("This is a test message")
wait(3)
NPApp.close()

Open Sikuli IDE and paste this into the IDE window.  You'll be unable to run it as is, though-- you need to supply at least two graphic elements: the Notepad menu bar and the Notepad About dialog OK button.  Highlight the text "Your Notepad Menu Bar" (including the quotes) in line 6 and, with Sikuli IDE still open, launch Notepad.  Press CTRL + SHIFT + 2-- you should see your screen "fade" slightly as Sikuli IDE enters its capture mode.  Click and drag the cross-hair across the menu bar items-- it should look something like this:


Capturing the menu bar

When you release the mouse button, the screenshot of the menu bar should automatically get inserted into the script:


Script with menu bar screenshot pasted

In Notepad, click Help in the menu bar, then "About Notepad" and repeat the steps above, this time replacing the "Your OK Button" text near the bottom of the script with a screenshot of the OK button in the About Notepad dialog.


OK button pasted in

Now try running the script (note that when the "Window is open!" message is displayed, the script is paused until you click the OK button).  Also make sure the path to Notepad is valid for your particular version of Windows (the script was written on a PC running Windows 7).

Now let's step through the lines of the script; if you're experiencing problems I'll try to highlight some "gotchas" as we go that might be the source.

In the first few lines of the script we're opening Notepad by creating a Sikuli App object-- the App.open() function does this using the path to your target application.

Sikuli's wait function (line 6) scans the screen for the existence of the indicated (text or graphic) element; the second parameter indicates how long Sikuli should wait (in seconds) for the element to appear before failing (you can use the constant FOREVER to specify that Sikuli should wait indefinitely)-- here we're giving Notepad five seconds to launch.  If you have a particularly slow machine where it seems to be taking a while for Notepad to launch, try upping this wait time parameter.  There's a corresponding function in Sikuli called waitVanish()-- this function behaves similarly to the wait function, but scans the screen for the absence of a given element (i.e., waits for it to vanish from the screen).  Other similar functions: find(), which (no surprise here) simply finds the given element onscreen without acting on it; click(), which we'll see below and finds and clicks on the specified element; and exists(), which checks for the presence of an element onscreen.

The popup() function (line 7) simply displays a message box with the indicated text-- as I mentioned previously, Sikuli will pause after the popup is displayed and wait for the user to click the OK button before continuing.  This can be useful for troubleshooting scripts or pausing the script to allow for a manual task.

In line 8, the menuBar region is retrieved using the getLastMatch() function-- this illustrates a useful side effect of many of Sikuli's basic functions, including the wait() function-- when the function is successful (e.g., the element that wait() is waiting for is detected) the location of the element is stored as a match that can be retrieved later via the getLastMatch() function.  So in the case of our script, once the wait() function in line 6 is successful, it's not necessary to re-find or otherwise identify the menu bar pattern (which can be a costly operation in terms of processing resources).

In line 10, the highlight() function is called for the menuBar region we just defined, which simply highlights the region for the given amount of time (in seconds).  This can be useful for troubleshooting scripts in instances where you're dynamically defining a region that's supposed to contain a specific element-- you can use the highlight to see what's actually included in the region.  A word of warning about the highlight() function, though:  I've seen cases where it may result in a redraw that "hides" some child windows (like drop-downs, for example).

In the next line, text recognition is used to identify the Help button in the menu bar and the identified area is clicked.  As I mentioned in my previous post, Sikuli IDE's text recognition can be a little finnicky sometimes.  If you seem to be having problems with this particular action in the script, try using a graphic element (an actual screenshot of the File button, captured as described above) instead of text.  Text recognition seems to work best for cases like this: a short string with a simple well-contrasted setting (e.g., black text on white).

One more important thing to point out in line 11 is the use of click() as a method of the menuBar region.  When functions are called as methods of regions, Sikuli only scans those regions, not the entire screen.  In fact, in line 6 of the script when we called the wait() function without specifying a region, it's called as a method of the default SCREEN region-- a region encompassing the entire screen.  Generally it's good practice to call these functions on regions instead of the entire screen since it's less costly to scan a subsection of the screen as opposed to the entire screen.

Line 13 defines a new region dynamically.  A Sikuli Region object's area is defined by four parameters-- x and y coordinates that define its upper left corner, and width and height parameters.  In this line in the script we're using the getLastMatch() function again, this time for the menuBar region.  The click() function in line 11, also called for the menuBar region, recorded the location of the "Help" text as a successful match; getLastMatch() returns it.  Using the match as our starting point, we can define a new region by getting the X and Y parameters of the "Help" text's region-- via getX() and getY()-- then making our new region 300 pixels wide by 100 pixels tall.  This line could cause problems depending on your computer's resolution-- if you find that the next line is failing (even after ruling out a text recognition failure) try making the region larger in this step-- it could be that the 300 by 100 pixel region is too small to encompass the About Notepad button.

The last few lines are self-explanatory or repeats of functions we've already seen: Sikuli finds and clicks on the text "About" in the newly defined dropDownRegion, then waits for 3 seconds as the About dialog is displayed.  After three seconds, the OK button is identified (via a screenshot this time, not text) and clicked. The text "This is a test message" is typed into Notepad's main window and finally, after a 3 second wait the app is closed using the App object created in line 3.

Of course, this is a very simple script and only illustrates the tip of the iceberg of Sikuli's power.  If you're interested in a more detailed look, I recommend checking out Sikuli's documentation (which I think is pretty well-written) here.

Sikuli IDE: An Overview


Switching gears quite a bit from my posts so far, in this post I'm going to give a quick overview of Sikuli IDE, a free UI automation tool.

On the face of it, UI automation testing seems like a pretty straightforward concept: mimic the actions that a user would perform while interacting with an application and confirm the application responds to those actions appropriately.  When you roll up your sleeves and start trying to do it, though, you quickly realize that it's often much easier said then done and there are lots of pitfalls and hurdles to overcome.  If you try to access UI controls programmatically (as objects), you may come up against custom controls that your tool of choice can't recognize , controls that aren't uniquely named, or seemingly infinite nested levels that make identifying and isolating individual controls difficult.  Tools that use UI manipulation (e.g., move cursor to coordinates x,y and click, tab to button 1, press the down arrow twice, etc.) can result in scripts that are obscure and/or diffictult to maintain.

Sikuli IDE (and other tools like it like Citratest and DeviceAnywhere) uses a technique that tries to smooth over some of these issues encountered with other strategies.  Basically, you capture GUI elements (buttons, links, etc.) in small graphic files; during script execution, Sikuli uses these captures to identify those elements onscreen and interact with them.  The strategy maintains some of the modularity of an object-oriented approach without any dependencies on the application's underlying architecture.

The documentation provided on the Project Sikuli site is decent, organized in a logical, straightforward manner by key objects.  As its name implies, Sikuli IDE also comes with its own IDE, where graphical "objects" are represented as images, making dealing with scripts very intuitive (Python/Jython is the language used for scripting in Sikuli).


Sikuli IDE

Of course, as you might expect from a tool that keys off graphics, Sikuli is less than ideal in situations where the UI is still unstable-- although the easy-to-use capture interface (basically, pressing CTRL + SHIFT + 2, then selecting the region you want to capture) makes it relatively easy to adjust to UI changes.  While Sikuli does claim to have some text recognition functionality, this seems to be in the very early stages and very unreliable at this point.


Sikuli's text recognition (sort of)

soapUI: A Simple Data-Driven Testing Technique

Note: If you experience any issues with webservicex.net or the periodic table service used in this post, or would just like to see another example, I've provided an alternative post covering pretty much the same material with a different service here.

For this post, I'd like to walk through an example project based on the periodic table service used in previous posts to illustrate the use of Groovy script.  You can download the project and a data file used by the project here. After unzipping the files, open soapUI and use File - Import Project to pull the project (xml file) into your workspace. You can put the TestData.csv file anywhere you want, really, but its default location in the script is a directory called C:\PerTableData-- you'll have to modify one line in the project if you want to put it somewhere else (note: to see one way to modify this project to use an Excel data source instead of CSV, see my post here).  Here's what the project tree should look like:


Example project tree

The four operations provided by the service (GetAtomicNumber, GetAtomicWeight, GetElementSymbol, and GetAtoms) are grouped into two separate test cases.  The GetAtoms operation has no request parameters and is in its own test case.  The three remaining operations are grouped together in the ElementSpecificRequests test case for several reasons.  All of them use a single element name as their only request parameter, and there's some overlap between the data returned by the operations (the GetAtomicNumber request returns atomic weights and symbols, which are also returned by the GetAtomicWeight and GetElementSymbol requests, respectively).  Ideally, of course, we'd like to test these three "element specific" operations using more than one symbol, so they're all candidates for data looping (repeating the test requests with different input parameters), too.

Our strategy for data looping is as follows:

1) Create a data file that provides request input parameters (element names, in this case) AND the expected response data for each input so we can perform content assertions.
2) Read in a single line of the file and populate test case properties with the data.
3) Perform our test requests and assertions using the test case properties we populated from the file.
4) Repeat steps 2 and 3 until we reach the end of our file (i.e., no more lines to read in).
5) End the test case.

The TestData.csv file is our source file for step 1.  I personally prefer using csv files for this purpose because they generally have a clear delimiter and they're easy to create and maintain in spreadsheet programs like Excel.  Here are the contents of TestData.csv; each line consists of an element's name, atomic number, symbol, atomic weight, and boiling point (you may notice some errors-- these are there to simulate test step "fails" later on):

Hydrogen,1,H,1.00797,20.400000000000002
Carbon,6,C,12.0115,51000
Oxygen,8,O,15.9994,90.2
Gold,79,Gd,196.967,3239
Uranium,92,U,238.03,4091

For data looping there's also a test case option that we want to change.  If you launch the Options dialog (right-click on a test case in the navigator and select Options from the context menu), you'll see that there's an option called "Abort on Error".  We want to make sure this is unchecked-- otherwise, execution will stop and move on to the next test case the first time an error occurs; obviously, we want the test case to continue looping even if an error occurs during one loop.

The Groovy script to start reading our csv file isn't contained in a test step-- instead, it's in the ElementSpecificRequests test case's Setup script.   At the test suite and test case levels, Setup and TearDown scripts are available-- the Setup script is run prior to any of the test steps of the test case (or prior to any of the test cases when its at the suite level), and the TearDown script is run after the last test step (or test case for suite level TearDown scripts) has completed.


The Test Case Editor with the Setup Script shown (note the button to show the TearDown Script, too)

Here's our Setup Script:

//Create a new filereader object, using the context variable so it can be used between test components
context.fileReader = new BufferedReader(new FileReader("C:\\PerTableData\\TestData.csv"))
//Read in the first line of the data file
firstLine = context.fileReader.readLine()
//Split the first line into a string array and assign the array elements to various test case properties
String[] propData = firstLine.split(",")
testCase.setPropertyValue("ElName",propData[0])
testCase.setPropertyValue("AtNum",propData[1])
testCase.setPropertyValue("Symbol",propData[2])
testCase.setPropertyValue("AtWeight",propData[3])
testCase.setPropertyValue("BoilPoint",propData[4])
//Rename request test steps for readability in the log; append the element name to the test step names
testCase.getTestStepAt(0).setName("GetAtomicNumber-" + propData[0])
testCase.getTestStepAt(1).setName("GetAtomicWeight-" + propData[0])
testCase.getTestStepAt(2).setName("GetElementySymbol-" + propData[0])

The first (non-comment) line creates a FileReader object to read our data input file (if you put the csv file into a location other than C:\PerTableData, you need to modify this line accordingly).  Note that our fileReader object is created as a property of the context variable.  The context variable is readily available in Setup and Teardown Scripts-- no need to declare it or otherwise initialize it.  It provides a way to share objects, variables, etc., across the relevant run context.  In this case, the fileReader object will be available across the test steps of our test case-- handy because we'll need to keep track of where we are in our file as we read in each line.

In the remaining lines of the script we read in the first line of our file (line 4), split the line into a string array (line 6), and then assign the elements of the array to our test case properties (lines 7 - 11) using setPropertyValue(), a method of the testCase object.  In lines 13 - 15 we're actually renaming the test steps by appending element names to their default names-- this is not necessary, really, but helpful for reading the run log; otherwise, we'd see identical test step names for each iteration.  Note the use of the testCase variable in lines 7 - 15; like the context variable, this is automatically available for use in the Setup Script.

After our Setup Script runs, our test requests are sent and verified against content assertions.  We use property expansions with our test case properties for request parameters and assertion criteria:


GetAtomicNumber test request with an assertion-- note the property expansions used for the ElementName element in the request and the AtomicNumber element in the assertion

So after reading in the first line of our data file, the GetAtomicNumber test request gets sent using "Hydrogen" as the ElementName; the response is checked to see if it contains "1".

After our test requests, the next step in the test case is the ReadNextLine Groovy test step.  Here's the script contained in this test step:

/*Read in the next line of the file
  We can use the same fileReader created in the Setup script because it
  was assigned to the context variable.*/
nextLine = context.fileReader.readLine()
/*If the end of the file hasn't been reached (nextLine does NOT equal null)
  split the line and assign new property values, rename test request steps,
  and go back to the first test request step*/
if(nextLine != null){
 String[] propData = nextLine.split(",")
 curTC = testRunner.testCase
 curTC.setPropertyValue("ElName",propData[0])
 curTC.setPropertyValue("AtNum",propData[1])
 curTC.setPropertyValue("Symbol",propData[2])
 curTC.setPropertyValue("AtWeight",propData[3])
 curTC.setPropertyValue("BoilPoint",propData[4])
 curTC.getTestStepAt(0).setName("GetAtomicNumber-" + propData[0])
 curTC.getTestStepAt(1).setName("GetAtomicWeight-" + propData[0])
 curTC.getTestStepAt(2).setName("GetElementSymbol-" + propData[0])
 testRunner.gotoStep(0)
}

Most of this should look pretty familiar-- mostly, we're doing the same things we did in our Setup script.  In line 4 we're reading in the next line in our data file.  If the next line doesn't equal null (which is what would be returned if we had reached the end of the file), we go through the same steps we went through before: splitting the line up into a string array (line 9), assigning the elements of that array to different test case properties (lines 11 - 15), and re-naming the request test steps (lines 16 - 18).

Two new things: the last line of the script sends test case execution back to the first step in our test case, the GetAtomicNumber request.  Also note the use of the testRunner variable.  This is another one of those variables like the context variable that's ready to use from the start.  The testRunner variable is important in soapUI scripting as it's used as an entry point to get to other test objects and control execution.  Case in point: the testCase variable that was available as a "built-in" variable in our Setup script is not immediately available in our Groovy test step.  Instead, in line 10 we use our testRunner variable to get a reference to the test case.  Once we have the test case, we can get its test steps and their properties and methods, and so on.

So execution passes back to the first test step and our test requests are re-run, this time using property values read in from the second line of our data file for request parameters and assertion criteria.  This loop continues until we reach the end of the file-- when this happens, the next line read in is null, so the if statement in line 8 of the ReadNextLine Groovy step fails and execution is never passed back to the first test step.

Finally, before the test case ends its TearDown script is run:

//Cleanup: rename test steps to their generic names and close the file reader
testCase.getTestStepAt(0).setName("GetAtomicNumber")
testCase.getTestStepAt(1).setName("GetAtomicWeight")
testCase.getTestStepAt(2).setName("GetElementSymbol")
context.fileReader.close()

In this script we're just renaming the request steps back to their default names and in the final line closing our data file for reading.

Try running the test suite-- double-click on the test suite node in the navigator to launch the TestSuite Editor, then click the Run button in the editor window.  You should see the progress bar for the ElementSpecificRequests test case fill up multiple times, turning red after an error is encountered.  Once the test cases have completed running, open up the TestSuite Log.


The TestSuite Log for our periodic table test suite (with run results)

We can see right away that something failed for the GetAtomicNumber test request-- and because we were modifying the test step names as we went, we can see it occurred for the carbon element.  If you want to see more information about the failure, you can simply click on the error to launch the Message Viewer.  Here you can see the request and response XML (including the values resulting from property expansions) to get a better idea of where the problem may be.  In this case, when we compare the response with our error above we can see that our assertion was expecting a boiling point of 51000 but our response returned a value of 5100.


Message Viewer response

That illustrates one way to perform data looping with the soapUI free version.  It should be noted that the commercial version of soapUI has features to make data looping much easier with less scripting required.  If data looping is a feature you intend to use heavily, it may be worth it to weigh the cost of the commercial version against potential savings in time spent on script development and maintenance.

soapUI: Other Test Steps

Now that we've taken a look at the SOAP Request test step and soapUI properties, let's take a look at some of the other test step types.


Available test steps (available in the free version of soapUI)

Request test steps are the workhorses of soapUI.  Don't let the name fool you-- soapUI isn't just limited to SOAP test requests; request test steps are also available for HTTP, AMF (Action Message Format), and REST (REpresentational State Transfer) services and JDBC database queries. The general idea is roughly the same for all of these request test steps: enter information for the service or database you're trying to test, create a test request, query, etc., and validate the response.


JDBC Test Request

See the soapUI starter articles for REST and JDBC for more information. One nifty extra feature (I think) worth mentioning is a web session recorder for HTTP/website testing; it allows you to click through a given web site and generates test steps based on your actions.

The MockResponse test step allows you to mimic a web service response, specifying the data returned via a script, a literal value, etc.


MockResponse

Like the test request test steps, MockResponses and MockServices are deep topics-- I'd recommend checking out soapUI's own MockServices help pages for more details.

Property-related test steps, not surprisingly, allow manipulation of properties.  The Properties test step is fairly straightforward, allowing you to introduce properties mid-test case.  Remember that if you reference the properties in a subsequent test step (which must be in the same test case), the property expansion syntax is "${PropertyTestStepName#PropertyName}".  For example, to reference ElName1 in the screenshot below in another test step later in the test case, you'd use the property expansion "${ElementProperties#ElName1}".


Properties Test Step

The Property Transfer test step lets you copy properties between test components.  When creating Transfers in a Property Transfer step you specify a source component (project, test step, etc.) and property and a target component and property.


Property Transfer Test Step

The screenshot above illustrates an important way that properties and property transfers or references can be used.  The source component in this case is a SOAP Request test step.  The request test step's response is exposed in soapUI as a property.  Using an XPATH expression, you can "drill down" further into the response to extract data from a specific XML node and transfer it to another component's property.  This functionality is useful in situations where the response from a test request may need to be used as an input parameter elsewhere (a common example cited in soapUI's help is getting a session token from a login response that needs to be passed in a subsequent request).

Flow control test steps include Delay, Run Test Case, and Conditional GoTo test steps. The Delay test step is self-explanatory-- you're just specifying a time period (in milliseconds) to pause during a test run. Run TestCase test steps allow you to call one test case from within another.  Configuring a Run TestCase step involves specifying the test case to run, providing properties to use when the test case is called, and specifying which properties you want returned from the test case being called.


Run TestCase


Run TestCase Options

In the Run TestCase test step shown above, the GetAtomicWeight test case is being called from within another test case.  AtomicWeight, ElNameIn, and AtomicWeightCData are all properties of the GetAtomicWeight test case.  When the test case is called, the ElNameIn property is set to "Helium", and the AtomicWeight property (in this case, extracted from a SOAP response) is returned from the test case run.  You can use property transfers or references to work with that returned AtomicWeight value (with the property expansion "${Run TestCase#AtomicWeight}", for example).

A practical example of where this might be useful is with a test suite where you have a login test case that retrieves a session token that's sent with other requests-- suppose you want to generate session tokens for different users through the course of your test suite.  You can just call the same login test case each time, passing in different usernames and passwords and returning their session tokens.

The Conditional GoTo test step evaluates a response (via an XPath expression) or property to control the test case flow-- if the response or property matches a specified condition, the selected target step is launched.


Conditional GoTo

In the screenshot above, the Conditional GoTo test step checks to see if the AtomicWeight property equals 1.00797; once this is confirmed to be the case, the test case skips to the Groovy Script test step designated in the Target step drop-down.  A practical use case for a Conditional GoTo might be a situation where you need to keep re-running a request until you get a specific response needed to continue on with the test case.

The last two test step types, Manual test steps and Groovy Script test steps, are their own animals.  Manual test steps allow for manual activities that need to be done during the course of executing test cases-- creating a Manual test step is as simple as specifying the manual actions that need to be taken and the expected result(s).  During test case execution, soapUI will pause while the actions are performed and the results are recorded.


Manual Test Step

Last (but certainly not least) is the Groovy Script test step, which should be pretty self-explanatory: you can write scripts in Groovy (described as "an object-oriented language alternative for the Java platform") to do pretty much whatever you want as part of your test suite.


Extremely unexciting Groovy Script Test Step

In the next post I'll look at Groovy scripting a little bit more, using it to perform simple data looping...

soapUI: Properties

I wanted to jump into a survey of the various test steps in this post, but quickly realized that an explanation of properties was necessary first.  Properties are a key concept in soapUI that can make test suites more flexible and modular.  You can define and reference your own custom properties (which you can think of as being akin to variables in soapUI) or use "built-in" properties to reference bits of information between test steps and test cases-- for example, referencing part of one request test step's response XML to use as an input parameter in another request test step.

Properties are available at multiple levels in soapUI, including the environment, system, global, project, test suite, test case, and test step levels.  System properties are visible in the Help - System Properties window.  Global properties can be viewed or set in the File - Preferences window.  Properties for other levels are visible in the Navigator panel when you click on the relevant node.


Project properties panel

In some cases, properties can also be loaded from or saved to external files (see the screenshot below of a test suite Custom Properties tab; the two rightmost icons at the top of the dialog are for loading and saving from external files).  Properties are saved in the file in propertyname=propertyvalue pairs; e.g.:

ElName=Mercury
ExpectedVal=80

Properties can be referenced (soapUI documentation calls this property expansion) from almost anywhere in soapUI-- in Contains assertions as the string to match, within a request, etc.  To reference a property, the general syntax is "${[#PropertyScope#]PropertyName[#XPathExpression]}".  PropertyScope can be "Env", "System", "Global", "Project", "TestSuite", and "TestCase" (in the screenshot above, the Hermes Config property references the user.home System property).  To reference properties from a preceding test step, the syntax is "${TestStepName#PropertyName[#XPathExpression]}-- note that the leading hash mark (#) before the scope is omitted when referencing a test step.  The test step where you're putting your reference and the test step being referenced must be within the same test case.  XPath expressions are optional and only valid where the property being referenced is valid XML (for test requests, the request and response are both made available as properties).

Let's go back to the test suite created in the last post for our Periodic Table web service.  Note that three of the four operations provided by the web service take an element name as their input parameter: GetAtomicNumber, GetAtomicWeight, GetElementSymbol.  Suppose we wanted to use the same element to test all three.  Instead of hard-coding an element into our requests, we can define a test suite level property and then use that property in all three of our test requests.

Let's try that now.  In the Navigator panel, click on the test suite node.  In the Properties panel (click the Properties button at the bottom of the Navigator panel if you don't see it), click on the "Custom Properties" tab, and add a new property called ElNameProp (via the left-most button at the top), giving it "Silver" as its value.


ElNameProp created at the test suite level

Now go into the GetAtomicNumber, GetAtomicWeight, and GetElementSymbol test requests and modify each to use our test suite's ElNameProp where an element name is required. Our reference expression is "${#TestSuite#ElNameProp}". Here's the reference in the GetAtomicNumber test request:


ElNameProp used in the GetAtomicNumber test request

Now try running each of the modified test requests-- the value "Silver" is plugged into each of the requests where we reference our ElNameProp property (you can see some of the appropriate response data for the GetAtomicNumber request in the screenshot above).

soapUI: Creating a Test Suite and Test Step

In my last post I provided a brief overview of soapUI, a popular tool for testing web services. In this post I'll go into a little more detail and cover some basic soapUI procedures and concepts.

As I mentioned in my previous post, soapUI can build a basic test suite from a WSDL.  For this mini-tutorial, I'll be using a web service based on the periodic table of elements that allows users to retrieve symbols, atomic weights, etc.  In soapUI, select File - New soapUI Project and enter a Project Name and the following Initial WSDL/WADL:

http://www.webservicex.net/periodictable.asmx?WSDL

Check the box to generate a test suite and click OK. You'll see soapUI automatically load definitions for our web service, after which it prompts the user for test suite creation options, which include selecting which operations you'd like to include and how you'd like them grouped in test cases. Just accept the defaults here (you may be prompted twice-- soapUI is building test suites for SOAP 1.1 and 1.2 bindings; you can use either for the following examples).

On the left-hand side of the window you'll see the workspace navigator; after a successful import you'll see interfaces and generated test suites for the web service.


soapUI Workspace Navigator

Using the default options, each operation will have its own test case consisting of a single SOAP request test step.  Let's actually build a test request and send it to the web service.  Drill down into the test suite for the periodic table web service and double-click on the GetElementSymbol test step to launch the test step editor.  The GetElementSymbol operation takes a single input parameter, an element name, and returns its symbol in the periodic table.  Modify the test request in the left pane of the editor so the ElementName parameter is "Silver" (replace the "?" between the ElementName tags with "Silver").  Then click the Submit Request button at the top of the window-- the leftmost button that looks like a Play button.  The test request is sent to the web service and you should see the response in the right pane.


SOAP Request Test Step

Now that we have a working SOAP request test step we'll want to create some assertions to validate our response data.  Click the Assertions button at the bottom of the soapUI window to expand the Assertions panel (displayed across the lower part of the window).  Click the Add Assertions button (leftmost button at the top of the Assertions panel) to launch the Add Assertion dialog-- here you can look through some of the different Assertion types that are available.


Add Assertion dialog

From the SLA section, select the Response SLA assertion type and click Add.  This assertion type confirms that the given test request receives a response within a specified period of time; you'll be prompted to specify the acceptable response time (in milliseconds) before the assertion is added.  Enter 10000 (10 seconds) here.

Add a Not SOAP Fault assertion (from the Compliance, Status and Standards section) to verify that the test response is not a SOAP Fault, and finally, add a Contains assertion from the Property Content section to validate response data.  Since this particular SOAP request is pretty straightforward with only one piece of data returned in the response (the element's symbol), the Contains assertion should be adequate.  You'll be prompted to enter a string to look for in your test response; enter text matching the response's Symbol element: "Ag" (the quotation marks should not be included).

For any subsequent runs of your test step, your SOAP response will be validated automatically against the added assertions with the pass/fail status of each indicated by "green light"/"red light" icons in the Assertions panel.


Assertions panel with pass/fail statuses indicated

The SOAP Request test step is just one test step type-- in the next post, we'll look at some of the others.

soapUI: An Overview

One of my favorite test apps I've encountered over the last few years is soapUI, a tool for testing SOAP web services. While it's also available in a Pro version for a fee, the freeware version is still pretty feature-rich with support for customization via Groovy, a Java-like scripting language. It's just a well-made program; when I first started working with it I'd frequently find myself wondering if there was an easier way to do X or Y-- and usually there was. It's almost spooky how well soapUI's makers have anticipated what users might want to do with it, and they're constantly adding meaningful functionality.

Setting up your basic test framework in soapUI is as easy as pointing it toward your web service's WSDL.  Using the provided WSDL information, it can generate a test suite with test requests built on available operations.


The main soapUI window

To test each request, you can define a set of pass/fail assertions.  SoapUI provides an impressive number of assertion types-- you can validate against response codes, confirm that responses are received within a set number of seconds, etc.  Data validation includes simple Contains/Not Contains tests that check for the presence (or absence) of specific data in responses or more flexible XQuery and XPath assertions that allow you to parse and analyze response XML data to ensure it matches expected results. 

I've found XQuery assertions particularly useful, especially when dealing with a mix of static and dynamic data in responses-- you can set up XQuery assertions in such a way that the static data (which is more suitable for automated verification) is isolated from the dynamic data and verify large data sets all at once.

It's also worth noting that a sister application for soapUI called loadUI was introduced within the last few years.  As you might guess, loadUI is a tool for performing load tests against a web service.  One of the selling points of loadUI is that you can use the same functional test requests developed in soapUI as your loadUI test requests.

Over the next few posts, I'll dig a little deeper into soapUI, including XQuery and XPath syntax for those who may not have worked with it before.

Introduction

With agile development methodologies growing in popularity, knowledge of some test automation tools and techniques is a useful thing for a software tester to have in his or her bag of tricks. In this blog I plan to cover some of the tools I've found useful and hopefully help software testers with little experience in automation get up and running quickly.

In the spirit of full disclosure, I'm not a developer by trade. Some of my scripting solutions may lack the elegance of a professional programmer-- but hopefully they should be good enough to get a job done when results are your main concern. As a software tester, I started out primarily using manual techniques but found myself wanting and needing to learn about automation as "agile" and "automation" became organizational buzz words-- I suspect many other software testers have found themselves in the same boat. The good news is that (in my opinion) learning about and implementing automation can be an interesting and rewarding exercise.

Presently, I don't envision any overarching organizational structure to the blog posts; basically, I hope each post by itself might provide some information that's useful to somebody. Over time, though, I'll probably revisit certain topics; I may group these blogs together as appropriate as they accumulate.