Automating SoapUI with Testrunner

In the SoapUI examples I've provided so far, test suites have been run through the UI.  At some point, however, you'll probably want to take your automation to the next step, running suites unattended and reviewing test results later.  We'll take a look at those topics over the next few posts, starting with SoapUI's testrunner.

Testrunner.bat typically resides in the bin sub-directory of SoapUI's program folder-- e.g., C:\Program Files (x86)\SmartBear\SoapUI-5.0.0\bin.  It allows you to run a SoapUI test suite from the command line with a variety of switches to set configuration options-- quite an extensive variety, in fact-- you can find the full list on the testrunner page on SoapUI's site.  Note that some of the switches listed here are only available in the Pro version of SoapUI.

Here's an example of a typical command sequence illustrating some of the more common switches you'll probably end up using:

"C:\Program Files (x86)\SmartBear\SoapUI-5.0.0\bin\testrunner.bat" -s"periodictableSoap TestSuite" -fC:\Temp\TestOutput -I -j -M C:\My Documents\MyPeriodicTableProject.xml

Going through each element of the sequence:

- "C:\Program Files (x86)\SmartBear\SoapUI-5.0.0\bin\testrunner.bat" : This is simply the path to testrunner.bat.  For those of you unfamiliar with using the command line, note the path is enclosed in quotations because it contains spaces.  Spaces normally separate elements of a command sequence; surrounding the path in quotations ensures the full path is treated as a single element.  Also, the full path is unnecessary if you've changed the working directory to the bin directory (i.e., you'd only have to specify "testrunner.bat"-- you can see this in the SoapUI TestRunner dialog below).

- -s"periodictableSoap TestSuite" : The -s switch precedes the name of the test suite within your project that you'd like to run-- as with the path to testrunner.bat, the name of the suite is enclosed in parentheses because it contains a space.

- -fC:\Temp\TestOutput : The -f switch precedes the path to the directory where test results will be saved.

- -I : This switch instructs testrunner to continue running even if an error is encountered.  Switches are case-sensitive (-i has a different meaning).

- -j : While SoapUI Pro supports a number of additional switches to generate different types of reports, the -j switch is available in the free version to generate JUnit-style xml reports.  These reports are saved to the directory specified with the -f switch above.

- -M : Separate from the JUnit-style report enabled by the -j switch, the -M switch saves a run log report to the results directory.

- C:\My Documents\MyPeriodicTableProject.xml : Finally, we specify the path to the project containing the suite to be run (no switch needed here).

Until you're comfortable with all of its switches and constructing a command sequence from scratch-- or just as a matter of preference-- you can also launch testrunner via the UI to let SoapUI generate a command sequence for you.  In the workspace tree, right-click on the name of the test suite for which you'd like to generate your command sequence (or the test project if you'd like to run multiple test suites in a given project) and select "Launch TestRunner" to bring up the configuration dialog.

Select your options from the various tabs in the dialog (the Basic and Reports tabs contain the more commonly used settings) and click the Launch button.  A command sequence corresponding to the selected configuration options is displayed near the top of the testrunner output (highlighted in the screenshot below).

Once you have this command line sequence it opens up a wide range of possibilities: you can incorporate it into your own batch files or Powershell scripts, integrate SoapUI test runs with an automation solution like Jenkins (incidentally, you can also use Jenkins to publish JUnit reports), schedule a run with Windows Task Scheduler, etc.

As mentioned above, the free version of SoapUI supports JUnit-style reports out of the box; however, it's possible to create some home-brewed reports using Groovy scripting.  We'll take a look at writing results to XLS files using the POI API in the next post.

Posts PDF and Downloads Page

Quite a while back (well over a year ago) I had a request from a reader to put my posts into a downloadable format.  I've finally gotten around to doing that; a PDF of almost all the posts published so far is now available on the new Downloads page (a link to the page is also available to the right).  I've re-ordered and grouped the posts in a way that I hope presents the information in a gradual, logical way.  Even so, it's difficult to translate blog posts to an offline format (obviously, they rely heavily on links).  I still hope some of you find it useful.  The sample projects referenced in many of the posts are also available for download on the same page.

XQuery in soapUI Part 2: Where and Order By Clauses

In the last post we took a look at the basic syntax of XQuery expressions in soapUI, including for and return clauses.  In this post we'll look at the optional where and order by clauses, which can add more complexity to expressions.

The XQuery where clause is very similar to the SQL where clause; it allows you to provide an additional filter on the base data set selected in your for clause.  In the last post we created an XQuery example using the Euro 2012 web service that retrieved data for each player returned in the AllPlayersNames test request.  Let's say we wanted to narrow down the list of players to include only members of the German national team.  We could adjust the query by adding a where clause just after the for clause:

In this case the where clause (where $x/m:sCountryName = 'Germany') whittles down the full set of tPlayerNames elements returned by the for clause, keeping only those that satisfy the condition of the where clause (in this case, those where the sCountryName child element equals "Germany").

Many of the same functions available in XPath expressions can also be used in XQuery expressions; the following is a valid where clause, for example:

where starts-with($x/m:sName/text(),'M')

The starts-with function takes two arguments and returns true if the first argument (in this case, the value of the sName element) begins with the second argument ("M").

In many cases, the choice to use a where clause may come down to preference; it's possible to achieve the same end result by using predicates with your for clause XPath expression.  This XPath expression in the for clause returns only German players (equivalent to the first where clause example):

for $x in //m:tPlayerNames[m:sCountryName='Germany']

Likewise, the following is equivalent to the second example, returning only those players with names starting with "M":

for $x in //m:tPlayerNames[starts-with(m:sName/text(),'M')]

For readability, however, you may find using where clauses preferable to the extended for clauses.

If a web service returns data in an arbitrary order (i.e., changing with each request), it presents a problem for XQuery assertions, which check for an exact match-- order absolutely matters.  For situations like these, an XQuery order by clause is necessary.  As you might expect, the order by clause is used to sort XQuery results by some node's value, ensuring results are always in the same order for evaluation by the assertion.  The order by clause is added before the return clause; here's an example of a complete XQuery expression including a for, where, order by, and return clause:

This example selects tPlayerNames elements from the response (the for clause), filters out only those elements with sName child element values that start with "M" (the where clause), orders the results by the value of the sName child element (the order by clause), and finally returns the values of the sName and sCountryName child elements for each tPlayerNames element, formatted within identifying tags (the return clause).

It's also possible to specify multiple values for the order by clause, establishing primary, secondary, etc. sort criteria.  For example, you can change the order by clause in the example above to the following to use sCountryName as the primary sort value and sName as the secondary sort value:

order by $x/m:sCountryName/text(), $x/m:sName/text()

The beginning of the results from this modified XQuery expression look like this:

As expected, results are sorted by country first, with players from the same country further sorted by name.

As usual when discussing XML-related topics, I highly recommend checking out W3Schools.com (available from the Useful Links section on the right) to find more information about using XQuery expressions.