Sikuli IDE: An Overview


Switching gears quite a bit from my posts so far, in this post I'm going to give a quick overview of Sikuli IDE, a free UI automation tool.

On the face of it, UI automation testing seems like a pretty straightforward concept: mimic the actions that a user would perform while interacting with an application and confirm the application responds to those actions appropriately.  When you roll up your sleeves and start trying to do it, though, you quickly realize that it's often much easier said then done and there are lots of pitfalls and hurdles to overcome.  If you try to access UI controls programmatically (as objects), you may come up against custom controls that your tool of choice can't recognize , controls that aren't uniquely named, or seemingly infinite nested levels that make identifying and isolating individual controls difficult.  Tools that use UI manipulation (e.g., move cursor to coordinates x,y and click, tab to button 1, press the down arrow twice, etc.) can result in scripts that are obscure and/or diffictult to maintain.

Sikuli IDE (and other tools like it like Citratest and DeviceAnywhere) uses a technique that tries to smooth over some of these issues encountered with other strategies.  Basically, you capture GUI elements (buttons, links, etc.) in small graphic files; during script execution, Sikuli uses these captures to identify those elements onscreen and interact with them.  The strategy maintains some of the modularity of an object-oriented approach without any dependencies on the application's underlying architecture.

The documentation provided on the Project Sikuli site is decent, organized in a logical, straightforward manner by key objects.  As its name implies, Sikuli IDE also comes with its own IDE, where graphical "objects" are represented as images, making dealing with scripts very intuitive (Python/Jython is the language used for scripting in Sikuli).


Sikuli IDE

Of course, as you might expect from a tool that keys off graphics, Sikuli is less than ideal in situations where the UI is still unstable-- although the easy-to-use capture interface (basically, pressing CTRL + SHIFT + 2, then selecting the region you want to capture) makes it relatively easy to adjust to UI changes.  While Sikuli does claim to have some text recognition functionality, this seems to be in the very early stages and very unreliable at this point.


Sikuli's text recognition (sort of)

soapUI: A Simple Data-Driven Testing Technique

Note: If you experience any issues with webservicex.net or the periodic table service used in this post, or would just like to see another example, I've provided an alternative post covering pretty much the same material with a different service here.

For this post, I'd like to walk through an example project based on the periodic table service used in previous posts to illustrate the use of Groovy script.  You can download the project and a data file used by the project here. After unzipping the files, open soapUI and use File - Import Project to pull the project (xml file) into your workspace. You can put the TestData.csv file anywhere you want, really, but its default location in the script is a directory called C:\PerTableData-- you'll have to modify one line in the project if you want to put it somewhere else (note: to see one way to modify this project to use an Excel data source instead of CSV, see my post here).  Here's what the project tree should look like:


Example project tree

The four operations provided by the service (GetAtomicNumber, GetAtomicWeight, GetElementSymbol, and GetAtoms) are grouped into two separate test cases.  The GetAtoms operation has no request parameters and is in its own test case.  The three remaining operations are grouped together in the ElementSpecificRequests test case for several reasons.  All of them use a single element name as their only request parameter, and there's some overlap between the data returned by the operations (the GetAtomicNumber request returns atomic weights and symbols, which are also returned by the GetAtomicWeight and GetElementSymbol requests, respectively).  Ideally, of course, we'd like to test these three "element specific" operations using more than one symbol, so they're all candidates for data looping (repeating the test requests with different input parameters), too.

Our strategy for data looping is as follows:

1) Create a data file that provides request input parameters (element names, in this case) AND the expected response data for each input so we can perform content assertions.
2) Read in a single line of the file and populate test case properties with the data.
3) Perform our test requests and assertions using the test case properties we populated from the file.
4) Repeat steps 2 and 3 until we reach the end of our file (i.e., no more lines to read in).
5) End the test case.

The TestData.csv file is our source file for step 1.  I personally prefer using csv files for this purpose because they generally have a clear delimiter and they're easy to create and maintain in spreadsheet programs like Excel.  Here are the contents of TestData.csv; each line consists of an element's name, atomic number, symbol, atomic weight, and boiling point (you may notice some errors-- these are there to simulate test step "fails" later on):

Hydrogen,1,H,1.00797,20.400000000000002
Carbon,6,C,12.0115,51000
Oxygen,8,O,15.9994,90.2
Gold,79,Gd,196.967,3239
Uranium,92,U,238.03,4091

For data looping there's also a test case option that we want to change.  If you launch the Options dialog (right-click on a test case in the navigator and select Options from the context menu), you'll see that there's an option called "Abort on Error".  We want to make sure this is unchecked-- otherwise, execution will stop and move on to the next test case the first time an error occurs; obviously, we want the test case to continue looping even if an error occurs during one loop.

The Groovy script to start reading our csv file isn't contained in a test step-- instead, it's in the ElementSpecificRequests test case's Setup script.   At the test suite and test case levels, Setup and TearDown scripts are available-- the Setup script is run prior to any of the test steps of the test case (or prior to any of the test cases when its at the suite level), and the TearDown script is run after the last test step (or test case for suite level TearDown scripts) has completed.


The Test Case Editor with the Setup Script shown (note the button to show the TearDown Script, too)

Here's our Setup Script:

//Create a new filereader object, using the context variable so it can be used between test components
context.fileReader = new BufferedReader(new FileReader("C:\\PerTableData\\TestData.csv"))
//Read in the first line of the data file
firstLine = context.fileReader.readLine()
//Split the first line into a string array and assign the array elements to various test case properties
String[] propData = firstLine.split(",")
testCase.setPropertyValue("ElName",propData[0])
testCase.setPropertyValue("AtNum",propData[1])
testCase.setPropertyValue("Symbol",propData[2])
testCase.setPropertyValue("AtWeight",propData[3])
testCase.setPropertyValue("BoilPoint",propData[4])
//Rename request test steps for readability in the log; append the element name to the test step names
testCase.getTestStepAt(0).setName("GetAtomicNumber-" + propData[0])
testCase.getTestStepAt(1).setName("GetAtomicWeight-" + propData[0])
testCase.getTestStepAt(2).setName("GetElementySymbol-" + propData[0])

The first (non-comment) line creates a FileReader object to read our data input file (if you put the csv file into a location other than C:\PerTableData, you need to modify this line accordingly).  Note that our fileReader object is created as a property of the context variable.  The context variable is readily available in Setup and Teardown Scripts-- no need to declare it or otherwise initialize it.  It provides a way to share objects, variables, etc., across the relevant run context.  In this case, the fileReader object will be available across the test steps of our test case-- handy because we'll need to keep track of where we are in our file as we read in each line.

In the remaining lines of the script we read in the first line of our file (line 4), split the line into a string array (line 6), and then assign the elements of the array to our test case properties (lines 7 - 11) using setPropertyValue(), a method of the testCase object.  In lines 13 - 15 we're actually renaming the test steps by appending element names to their default names-- this is not necessary, really, but helpful for reading the run log; otherwise, we'd see identical test step names for each iteration.  Note the use of the testCase variable in lines 7 - 15; like the context variable, this is automatically available for use in the Setup Script.

After our Setup Script runs, our test requests are sent and verified against content assertions.  We use property expansions with our test case properties for request parameters and assertion criteria:


GetAtomicNumber test request with an assertion-- note the property expansions used for the ElementName element in the request and the AtomicNumber element in the assertion

So after reading in the first line of our data file, the GetAtomicNumber test request gets sent using "Hydrogen" as the ElementName; the response is checked to see if it contains "1".

After our test requests, the next step in the test case is the ReadNextLine Groovy test step.  Here's the script contained in this test step:

/*Read in the next line of the file
  We can use the same fileReader created in the Setup script because it
  was assigned to the context variable.*/
nextLine = context.fileReader.readLine()
/*If the end of the file hasn't been reached (nextLine does NOT equal null)
  split the line and assign new property values, rename test request steps,
  and go back to the first test request step*/
if(nextLine != null){
 String[] propData = nextLine.split(",")
 curTC = testRunner.testCase
 curTC.setPropertyValue("ElName",propData[0])
 curTC.setPropertyValue("AtNum",propData[1])
 curTC.setPropertyValue("Symbol",propData[2])
 curTC.setPropertyValue("AtWeight",propData[3])
 curTC.setPropertyValue("BoilPoint",propData[4])
 curTC.getTestStepAt(0).setName("GetAtomicNumber-" + propData[0])
 curTC.getTestStepAt(1).setName("GetAtomicWeight-" + propData[0])
 curTC.getTestStepAt(2).setName("GetElementSymbol-" + propData[0])
 testRunner.gotoStep(0)
}

Most of this should look pretty familiar-- mostly, we're doing the same things we did in our Setup script.  In line 4 we're reading in the next line in our data file.  If the next line doesn't equal null (which is what would be returned if we had reached the end of the file), we go through the same steps we went through before: splitting the line up into a string array (line 9), assigning the elements of that array to different test case properties (lines 11 - 15), and re-naming the request test steps (lines 16 - 18).

Two new things: the last line of the script sends test case execution back to the first step in our test case, the GetAtomicNumber request.  Also note the use of the testRunner variable.  This is another one of those variables like the context variable that's ready to use from the start.  The testRunner variable is important in soapUI scripting as it's used as an entry point to get to other test objects and control execution.  Case in point: the testCase variable that was available as a "built-in" variable in our Setup script is not immediately available in our Groovy test step.  Instead, in line 10 we use our testRunner variable to get a reference to the test case.  Once we have the test case, we can get its test steps and their properties and methods, and so on.

So execution passes back to the first test step and our test requests are re-run, this time using property values read in from the second line of our data file for request parameters and assertion criteria.  This loop continues until we reach the end of the file-- when this happens, the next line read in is null, so the if statement in line 8 of the ReadNextLine Groovy step fails and execution is never passed back to the first test step.

Finally, before the test case ends its TearDown script is run:

//Cleanup: rename test steps to their generic names and close the file reader
testCase.getTestStepAt(0).setName("GetAtomicNumber")
testCase.getTestStepAt(1).setName("GetAtomicWeight")
testCase.getTestStepAt(2).setName("GetElementSymbol")
context.fileReader.close()

In this script we're just renaming the request steps back to their default names and in the final line closing our data file for reading.

Try running the test suite-- double-click on the test suite node in the navigator to launch the TestSuite Editor, then click the Run button in the editor window.  You should see the progress bar for the ElementSpecificRequests test case fill up multiple times, turning red after an error is encountered.  Once the test cases have completed running, open up the TestSuite Log.


The TestSuite Log for our periodic table test suite (with run results)

We can see right away that something failed for the GetAtomicNumber test request-- and because we were modifying the test step names as we went, we can see it occurred for the carbon element.  If you want to see more information about the failure, you can simply click on the error to launch the Message Viewer.  Here you can see the request and response XML (including the values resulting from property expansions) to get a better idea of where the problem may be.  In this case, when we compare the response with our error above we can see that our assertion was expecting a boiling point of 51000 but our response returned a value of 5100.


Message Viewer response

That illustrates one way to perform data looping with the soapUI free version.  It should be noted that the commercial version of soapUI has features to make data looping much easier with less scripting required.  If data looping is a feature you intend to use heavily, it may be worth it to weigh the cost of the commercial version against potential savings in time spent on script development and maintenance.

soapUI: Other Test Steps

Now that we've taken a look at the SOAP Request test step and soapUI properties, let's take a look at some of the other test step types.


Available test steps (available in the free version of soapUI)

Request test steps are the workhorses of soapUI.  Don't let the name fool you-- soapUI isn't just limited to SOAP test requests; request test steps are also available for HTTP, AMF (Action Message Format), and REST (REpresentational State Transfer) services and JDBC database queries. The general idea is roughly the same for all of these request test steps: enter information for the service or database you're trying to test, create a test request, query, etc., and validate the response.


JDBC Test Request

See the soapUI starter articles for REST and JDBC for more information. One nifty extra feature (I think) worth mentioning is a web session recorder for HTTP/website testing; it allows you to click through a given web site and generates test steps based on your actions.

The MockResponse test step allows you to mimic a web service response, specifying the data returned via a script, a literal value, etc.


MockResponse

Like the test request test steps, MockResponses and MockServices are deep topics-- I'd recommend checking out soapUI's own MockServices help pages for more details.

Property-related test steps, not surprisingly, allow manipulation of properties.  The Properties test step is fairly straightforward, allowing you to introduce properties mid-test case.  Remember that if you reference the properties in a subsequent test step (which must be in the same test case), the property expansion syntax is "${PropertyTestStepName#PropertyName}".  For example, to reference ElName1 in the screenshot below in another test step later in the test case, you'd use the property expansion "${ElementProperties#ElName1}".


Properties Test Step

The Property Transfer test step lets you copy properties between test components.  When creating Transfers in a Property Transfer step you specify a source component (project, test step, etc.) and property and a target component and property.


Property Transfer Test Step

The screenshot above illustrates an important way that properties and property transfers or references can be used.  The source component in this case is a SOAP Request test step.  The request test step's response is exposed in soapUI as a property.  Using an XPATH expression, you can "drill down" further into the response to extract data from a specific XML node and transfer it to another component's property.  This functionality is useful in situations where the response from a test request may need to be used as an input parameter elsewhere (a common example cited in soapUI's help is getting a session token from a login response that needs to be passed in a subsequent request).

Flow control test steps include Delay, Run Test Case, and Conditional GoTo test steps. The Delay test step is self-explanatory-- you're just specifying a time period (in milliseconds) to pause during a test run. Run TestCase test steps allow you to call one test case from within another.  Configuring a Run TestCase step involves specifying the test case to run, providing properties to use when the test case is called, and specifying which properties you want returned from the test case being called.


Run TestCase


Run TestCase Options

In the Run TestCase test step shown above, the GetAtomicWeight test case is being called from within another test case.  AtomicWeight, ElNameIn, and AtomicWeightCData are all properties of the GetAtomicWeight test case.  When the test case is called, the ElNameIn property is set to "Helium", and the AtomicWeight property (in this case, extracted from a SOAP response) is returned from the test case run.  You can use property transfers or references to work with that returned AtomicWeight value (with the property expansion "${Run TestCase#AtomicWeight}", for example).

A practical example of where this might be useful is with a test suite where you have a login test case that retrieves a session token that's sent with other requests-- suppose you want to generate session tokens for different users through the course of your test suite.  You can just call the same login test case each time, passing in different usernames and passwords and returning their session tokens.

The Conditional GoTo test step evaluates a response (via an XPath expression) or property to control the test case flow-- if the response or property matches a specified condition, the selected target step is launched.


Conditional GoTo

In the screenshot above, the Conditional GoTo test step checks to see if the AtomicWeight property equals 1.00797; once this is confirmed to be the case, the test case skips to the Groovy Script test step designated in the Target step drop-down.  A practical use case for a Conditional GoTo might be a situation where you need to keep re-running a request until you get a specific response needed to continue on with the test case.

The last two test step types, Manual test steps and Groovy Script test steps, are their own animals.  Manual test steps allow for manual activities that need to be done during the course of executing test cases-- creating a Manual test step is as simple as specifying the manual actions that need to be taken and the expected result(s).  During test case execution, soapUI will pause while the actions are performed and the results are recorded.


Manual Test Step

Last (but certainly not least) is the Groovy Script test step, which should be pretty self-explanatory: you can write scripts in Groovy (described as "an object-oriented language alternative for the Java platform") to do pretty much whatever you want as part of your test suite.


Extremely unexciting Groovy Script Test Step

In the next post I'll look at Groovy scripting a little bit more, using it to perform simple data looping...

soapUI: Properties

I wanted to jump into a survey of the various test steps in this post, but quickly realized that an explanation of properties was necessary first.  Properties are a key concept in soapUI that can make test suites more flexible and modular.  You can define and reference your own custom properties (which you can think of as being akin to variables in soapUI) or use "built-in" properties to reference bits of information between test steps and test cases-- for example, referencing part of one request test step's response XML to use as an input parameter in another request test step.

Properties are available at multiple levels in soapUI, including the environment, system, global, project, test suite, test case, and test step levels.  System properties are visible in the Help - System Properties window.  Global properties can be viewed or set in the File - Preferences window.  Properties for other levels are visible in the Navigator panel when you click on the relevant node.


Project properties panel

In some cases, properties can also be loaded from or saved to external files (see the screenshot below of a test suite Custom Properties tab; the two rightmost icons at the top of the dialog are for loading and saving from external files).  Properties are saved in the file in propertyname=propertyvalue pairs; e.g.:

ElName=Mercury
ExpectedVal=80

Properties can be referenced (soapUI documentation calls this property expansion) from almost anywhere in soapUI-- in Contains assertions as the string to match, within a request, etc.  To reference a property, the general syntax is "${[#PropertyScope#]PropertyName[#XPathExpression]}".  PropertyScope can be "Env", "System", "Global", "Project", "TestSuite", and "TestCase" (in the screenshot above, the Hermes Config property references the user.home System property).  To reference properties from a preceding test step, the syntax is "${TestStepName#PropertyName[#XPathExpression]}-- note that the leading hash mark (#) before the scope is omitted when referencing a test step.  The test step where you're putting your reference and the test step being referenced must be within the same test case.  XPath expressions are optional and only valid where the property being referenced is valid XML (for test requests, the request and response are both made available as properties).

Let's go back to the test suite created in the last post for our Periodic Table web service.  Note that three of the four operations provided by the web service take an element name as their input parameter: GetAtomicNumber, GetAtomicWeight, GetElementSymbol.  Suppose we wanted to use the same element to test all three.  Instead of hard-coding an element into our requests, we can define a test suite level property and then use that property in all three of our test requests.

Let's try that now.  In the Navigator panel, click on the test suite node.  In the Properties panel (click the Properties button at the bottom of the Navigator panel if you don't see it), click on the "Custom Properties" tab, and add a new property called ElNameProp (via the left-most button at the top), giving it "Silver" as its value.


ElNameProp created at the test suite level

Now go into the GetAtomicNumber, GetAtomicWeight, and GetElementSymbol test requests and modify each to use our test suite's ElNameProp where an element name is required. Our reference expression is "${#TestSuite#ElNameProp}". Here's the reference in the GetAtomicNumber test request:


ElNameProp used in the GetAtomicNumber test request

Now try running each of the modified test requests-- the value "Silver" is plugged into each of the requests where we reference our ElNameProp property (you can see some of the appropriate response data for the GetAtomicNumber request in the screenshot above).