SoapUI: Building JsonPath Expressions

The 5.2 release of SoapUI OS contains some significant changes.  Most obviously, there's been an overhaul of its basic look with new icons and some high-level actions (creating new test suites, importing suites, etc.) added to the main screen.  For the most part, these graphical changes are for ease-of-use and aesthetics; very little has really changed with basic workflows and 5.2 should be compatible with test suites created in pre-5.2 versions.  But in keeping with SoapUI's recent trend of improved support for non-SOAP services, several new JsonPath-based assertion types have also been added:

1) JsonPath Match - Verifies the value specified by a JsonPath expression matches some expected value

2) JsonPath Existence Match - Verifies the element specified by a JsonPath expression exists in the target JSON

3) JsonPath Count Match - Verifies the number of matching instances for a given JsonPath expression (if the expression evaluates to an array, the count is the length of the array) equals some expected value

4) JsonPath Regex Match - Similar to the basic JsonPath Match assertion, but the value specified by the JsonPath expression is matched to a regular expression

For those of you unfamiliar with it, JSON stands for JavaScript Object Notation.  Like XML, it provides a standardized way to represent complex data across different operating systems, architectures, etc.  Here's a typical JSON data block:
{"response":
   {
      "library":[
         {
            "title":"War and Peace",
            "author":"Tolstoy, Leo",
            "pages":1296,
            "tags":["literature","Russian","classic"]
         },
         {
            "title":"Harry Potter and the Chamber of Secrets",
            "author":"Rowling, J.K.",
            "pages":341,
            "tags":["fiction","fantasy"]
         },
         {
            "title":"In the Garden of Beasts",
            "author":"Larson, Erik",
            "pages":448,
            "tags":["non-fiction","history"]
         }
      ],
      "requestId":12345,
      "requestStatus":"OK"
   }      
}
This JSON defines an object (enclosed in braces in JSON) called response that consists of library, requestId, and requestStatus properties.  The library property is an array (enclosed in squared brackets) of objects representing books.  Each book object has a title, author, pages, and tags property; the tags property is itself another array of strings.

JsonPath is JSON's equivalent of XPath-- it provides a simple way to reference a particular value in JSON.  You can read a detailed description of JsonPath at http://goessner.net/articles/JsonPath, but I'll present some quick starting rules here with examples using the JSON data above.  JsonPath actually supports two different basic syntaxes, one using bracketed property names to navigate through successive levels of data (e.g., ['parent']['child']), and another using parent.child dot notation syntax similar to what's used in most programming languages.  For the sake of brevity, I'll focus on dot notation syntax in this post.

DescriptionExample ExpressionExample Result
To get the value of an object's property, separate the property name from the object name (or the portion of the JsonPath expression designating the object) with a dot-- again, if you're familiar with dot notation in programming, it's pretty much the same idea.response.requestStatusOK - The response object's requestStatus value
Zero-based indexing using brackets indicates a particular element in a JSON array.  Note the use of zero-based indexing (i.e., representing the first item with index 0, the second with index 1, etc.) is a key difference from XPath's one-based indexing.response.library[0].authorTolstoy, Leo - The author property of the first object in the library array
Using * instead of an index number with an array returns all elements in the array.response.library[*].title[War and Peace, Harry Potter and the Chamber of Secrets, In the Garden of Beasts] - A JSON array of every title in the library array
The JSON data root is designated by $, although this is optional in expressions that explicitly start with the first element of the target JSON data (the examples given above safely omit the $ since they start with the root response element).$.response.requestStatusOK - Equivalent to the first example; the requestStatus property of the response object
A pair of dots indicates a recursive search for the following property; i.e., the property can be found at any level beneath the property preceding the dots.  This is equivalent to double slashes in XPath.$..requestId12345 - The response object's requestId property (note we didn't have to explicitly designate the response object here)
You can access the length property of an array to determine the index of its last (or some position relative to the last) member.  In the example expression, @ represents the last evaluated element-- in this case, the library array-- in the bracketed expression.  Remember that because of zero-based indexing, the index of the last member of an array is one less than the array's length, so we subtract 1 from the length to find its index.  JsonPath is very particular about brackets and parentheses; the parentheses around the @.length - 1 expression may seem optional, but omitting them will result in an "invalid expression" error.$..library[(@.length - 1)].authorLarson, Erik - The author property of the last member in the library array.
You can also build basic predicate statements within squared brackets to filter on specific criteria.  The ? enclosed in brackets indicates a predicate, followed by a test expression in parentheses.  Within the predicate expression, the @ character acts as a placeholder for each array member being evaluated by the predicate.  So in this example expression we're starting with the result array, then evaluating each member's name property and returning those members where the title property matches 'War and Peace' (note the use of == instead of = to check for equality here).  Finally, for each array member that passed the predicate test, we get the value of its pages property.result.library[?(@.pages > 400)].title[War and Peace, In the Garden of Beasts] - The title property of every book object with a pages property greater than 400

If you'd like to play around with JSON expressions yourself, you can copy and paste the JSON data above into the JsonPath expression tester at http://jsonpath.curiousconcept.com.  One thing to keep in mind when working with the JsonPath expression tester: string values returned by JsonPath expressions in SoapUI are not enclosed in quotes, as they are with the JsonPath tester (the examples above show results as they appear in SoapUI).

SoapUI: Using FileAppender for Logging

In writing SoapUI test scripts, you may come across a situation where you need some additional logging functionality; for example, writing information to a file aside from the default SoapUI log for review after running a script via testrunner.  You could explicitly create and manage your own text file, but you may find it simpler to leverage the existing Logger object (via the log variable) to send logged messages to another file using the FileAppender class.

A FileAppender can forward messages sent to the log to a separate file, using formatting options to label messages with dates, logging levels, etc.  Here's an example setup script demonstrating how to set up a FileAppender to write to a simple text file (you can download a starter project containing all this code here):

import org.apache.log4j.FileAppender
import org.apache.log4j.PatternLayout

//Define the layout to use
def pLayout = new PatternLayout("%d{DATE} : %m%n")
//Create FileAppender in context using layout we just defined
context.customLogger = new FileAppender(pLayout,"C:\\Temp\\CustomLog\\CustomLog.txt", false)
//Add the appender to the existing log
log.addAppender(context.customLogger)

Remember, there's no need to add any additional jars to SoapUI's ext folder in this case; we're using the log4j package that's already part of its libraries.  However, we do need to import the FileAppender and PatternLayout classes at the top of our script.

After the import statements, a new PatternLayout is created for use with the FileAppender:

def pLayout = new PatternLayout("%d{DATE} : %m%n")

The string that's passed into the PatternLayout constructor ("%d{DATE} : %m%n") is used to describe how we'd like information formatted in the log file.  For a detailed list of the special formatting indicators (generally demarcated by the % character), see the javadoc for the log4j PatternLayout class.  In the meantime, here's a table showing some example patterns and how the line log.info("Simple Test Message One") is rendered in a custom log (starting with the pattern used in our script):

PatternOutputNotes
%d{DATE} : %m%n
14 Sep 2015 21:51:21,223 : Simple Test Message One
- %d indicates a date, with {DATE} specifying a log4j pre-canned date format
- %m indicates where the actual message content will be shown
- %n indicates a new line character
%-15d{ABSOLUTE}%p : %m%n
16:25:33,000   INFO : Simple Test Message One
- {ABSOLUTE} is used with %d to specify another log4j pre-canned date format-- this one prints only the time component of the {DATE} default format used in the first example.
- The -15 at the beginning of the date pads the expression, reserving 15 characters for the date and right-padding with spaces if the date is less than 15 characters long.
- %p indicates the priority level of the message (e.g., INFO, DEBUG, etc.).
%26d{ISO8601} : %m%n
   2015-09-19 16:50:08,095 : Simple Test Message One
- {ISO8601} is the last of the log4j pre-canned date formats, printing the date portion with numeric year, month, and day. ISO8601 is also the default format used if none is specified.
- The 26 at the beginning of the date format expression pads the date expression like -15 in the previous example, but the positive value pads the left instead of the right (note the leading spaces in the output).
%d{MM-dd-yyyy HH:mm:ss.SSS} : %.10m%n
09-19-2015 17:31:10.976 : essage One
- In this case we're using a custom format for the date; this one prints the date with numeric month, then day, then year, then prints the time using a decimal point as the divider between seconds and nanoseconds. You can find out more about the date and time pattern characters to use for a custom date format here. According to the log4j javadocs, using a custom format like this could result in a performance hit; they encourage you to use the pre-canned formats instead.
- The .10 before the message specifier is another spacing indicator; this one specifies that if the message is longer than 10 characters, it should be truncated from the front (which is why you only see the last 10 characters in the output).

Once we have our PatternLayout created, we can create our FileAppender, using the PatternLayout as an argument in its constructor.  We put it in context to manage disposing its resources later on in the tear down script:

context.customlogger = new FileAppender(pLayout, "C:\\Temp\\CustomLog\\CustomLog.txt", false)

The first two arguments should be self-explanatory: the first is the layout we want the FileAppender to use and the second is a string giving the path and name of the text file where we want the FileAppender to forward messages-- feel free to change this path as you see fit.  Remember that "\" has a special meaning in string in Java; that's why we have to use the escape sequence "\\" here instead.  The final argument specifies whether we want the messages appended to the text file.  When set to true, any content in the text file generated in previous runs is preserved and information from this run is added to the end of the file.  In this example we're setting the argument to false, so each time we run our suite a new log is created.

The final line in the setup script registers the appender with the default SoapUI log.

log.addAppender(context.customLogger)

Once everything is configured and the appender is registered with the log object, there's nothing special you have to do to send messages to this custom log file-- anything sent to the log within your script is sent to the FileAppender's underlying stream.

As indicated in the table above, the line log.info("Simple Test Message One") in our script results in this output in the custom log file:

14 Sep 2015 21:51:21,223 : Simple Test Message One

Within our tear down script (or whenever we're done with the appender), we have to release the resources being used by the appender, closing the underlying file and the appender itself:

context.customLogger.closeFile()
context.customLogger.close()

I encourage you to play around with the provided starter project to see what can be done with a file appender.  In particular, try making changes to the pattern layout string to see how it controls formatting and affects the custom log's output. 

Jenkins: An Example Project

In this post I'll provide a more detailed example of setting up a project in Jenkins, albeit on a smaller scale than what you would expect to encounter in a real-world situation.  I'm also not going into the specifics of some of the steps (for example, I'm not providing detailed explanations of the batch commands to run MSBuild or NUnit); my primary objective is to give you some idea of the general workflow of a Jenkins project and how its different components can fit together.

The particular scenario covered here involves a small solution consisting of a C# .NET application that's a basic calculator with several methods for basic mathematical operations, along with a library of NUnit tests that check those methods.  The code is maintained in a local Subversion repository.  In my walk-through I'll create a Jenkins job to handle building, testing, and reporting test results whenever there's a change to the SVN repository.

Clicking "New Item" brings up the page below.  In this case, I'm creating a Freestyle Project named "Silly Calc Project".


After clicking OK I'm presented with the main configuration screen for the project.  The page is broken up into a few sections by default.  For this example, the first section where I'll make changes is the Source Code Management section, where I can specify information to integrate a version control system with the project.  CVS and SVN repositories are supported by default; for other version control systems like git, search available Jenkins plug-ins using appropriate keywords.  Here I'm specifying that my repository is an SVN repository and providing its path.  This example uses the svn protocol, but Jenkins supports other protocols including http and https (see this page for more info about supported URL formats).


The Build Triggers section of the configuration page allows me to specify when Jenkins will start processing a build-- or really any job, since a project does not necessarily require a build step.  Default (included with Jenkins "out of the box") build triggers allow me to use other projects as a cue for when to start building this one, kick off the build on a regular scheduled basis, or build when a change is detected in the version control system.  Settings for the third option are illustrated here.


The Build Periodically and Poll SCM options use a syntax based on cron to specify how frequently a build/process is triggered (Build Periodically) or the version control system is polled for changes (Poll SCM).  Jenkins provides some inline help on this syntax (accessed via the help icon next to the text box where the cron expression is entered) or you can Google "cron" to find other explanations online-- Jenkins itself has a link to this Wikipedia entry.  Expressions consists of five parts, corresponding to (from left to right) minutes in an hour, hours in a day based on a 24-hour clock, days in a month, months, and days of the week.  The particular expression above instructs Jenkins to poll SVN from Monday through Friday (corresponding to "1-5" specified for days of the week) in five minute intervals ("H/5" specified for the minutes portion of the expression-- see the suggested cron explanations for details on the meaning of "H" in this case), with no additional restrictions based on hour, day of the month, or month in the year (specified by the "*" wild card character for those parts of the expression).

Now that I've specified when I want my job to start, I have to specify exactly what I want Jenkins to do in the Build configuration section.  When I click on the "Add build step" button, I see build options for Ant and Maven (Java-based build utilities) along with the ability to add shell or Windows batch commands.  This provides a lot of flexibility with Jenkins-- almost any task you can run via a batch command can be performed as part of a project.  Since my sample project was built using C# .NET I need to use a batch command to compile using MSBuild.


It's possible to add more than one batch command, and in this example I still need to add a command to run my NUnit tests once the build is complete.


Note how the paths to the solution (in the MSBuild command-- SillyCalc.sln) and the test library (in the NUnit command-- SillyCalcTest\bin\debug\SillyCalcTest.dll) are not fully qualified.  Each Jenkins project has an associated workspace folder.  Many commands and settings-- like the paths in these Windows batch commands-- are evaluated relative to this workspace folder (Jenkins Help will clearly specify when this is the case).

Finally, I want to publish the results of my tests, a Post-build Action.  Unfortunately, if I click on the "Add post-build action" button with a new Jenkins installation, I see that it's possible to publish JUnit reports, but there's no option available for NUnit reports.  This is one of those cases where the huge library of plug-ins pays off.  A quick search on "NUnit" in the Available tab of the Plugin Manager finds the NUnit plugin.  Once installation of the plug-in is finished, "Publish NUnit test result report" magically appears as an available Post-build Action in the project configuration page.


As with the batch commands above, the path to the NUnit report is relative to the workspace folder.  Given my settings, the report is generated in the workspace folder itself, so I only have to specify the file name.

So now that my Jenkins project is set up, it should detect any errors (as determined by my NUnit tests) that might be introduced in my code.  As an example, say I inadvertently change my calculator's add() method, replacing a + operator with a * operator, and check in the "bad" change to my SVN repository.  After a few minutes Jenkins will 1) poll SVN and recognize that a change was checked in, 2) check out the code in the repository, 3) run MSBuild on the solution, 4) run my NUnit tests on the newly generated build, and 5) publish test results so they're visible within Jenkins:


Again, this is a simple example; in truth this type of error probably would have been caught quickly enough with manual unit testing for a small application of this size.  But extrapolate this scenario out to a large project with many moving parts and interdependencies and many developers or even teams of developers, and you may see the potential benefits of using Jenkins for automation.