Data-Driven Testing Example 2: Holidays Web Service

In this post I'll walk through a sample soapUI project illustrating a data-driven testing technique that leverages test case properties and some light scripting.  This covers the same ground as my previous post on a simple data-driven testing technique, with a few minor differences; for example, this project uses XPath Match assertions where the original post used simple Contains assertions.

You can download the sample project and its accompanying CSV file (used as the source of our inputs and expected outputs) here in a single zipped file.  Download and unzip the files and import the project into soapUI.  You can place the CSV file wherever you'd like, although by default the script expects to find it in a directory called C:\Temp; you'll have to make a corresponding modification to the script if you'd like to put it somewhere else.

The service provides operations that return data related to holidays for the US and the UK; I've created test cases in the project for four of the operations, but it's the GetHolidayDate test case that we'll focus on in terms of data-driven testing.  GetHolidayDate takes a country, year, and holiday code (two of the other services show available country and holiday codes) and returns the date of the given holiday.

The holidays.csv file is our source file; each row of the file consists of a set of input parameters (country, holiday code, and year, in that order) along with our expected result-- the date corresponding to the input parameters.  Here are the contents of the file (just a sampling of all the available countries and holidays):

US,NEW_YEARS,2014,2014-01-01T00:00:00
GBEAW,BURNS_NIGHT,2014,2014-01-25T00:00:00
GBSCT,EMMELINE_PANKHURST,2010,2010-04-17T00:00:00
GBNIR,HOLOCAUST,2000,2000-01-27T00:00:00
US,EASTER,1980,1980-04-06T00:00:00
GBEAW,GUY_FAWKES,2008,2008-11-05T00:00:00
GBSCT,PALM_SUN,2013,2013-03-24T00:00:00
GBNIR,ST_PATRICKS_DAY,2005,2005-03-17T00:00:00
US,FLAG,2004,2004-06-14T00:00:00

We also have several test case properties defined:

Initially, these properties have no assigned values-- we'll fill those in with values from our source file as we iterate over the test case, doing the following:

1) Read a single line of the file, assigning the values in the line to corresponding test case properties.
2) Use property expansion to generate a request using the country, holiday code, and holiday year values we assigned to test case properties in step 1.
3) Use property expansion with an XPath assertion to confirm the response data returns the expected date, which was also assigned to a test case property in step 1.
4) Repeat steps 1 through 3 until the end of the file is reached.

The code to initialize our test data (opening the file for reading and processing in the first line of data) is placed in the test case setup script:

//Create a new BufferedReader object, using the context variable so it can be used between test components
context.sourceFile = new BufferedReader(new FileReader("C:\\Temp\\holidays.csv"))
//Read in the first line of the data file
def firstLine = context.sourceFile.readLine()
//Split the first line into a string array and assign the array members to various test case properties
def propData = firstLine.split(",")
testCase.setPropertyValue("holCountry",propData[0])
testCase.setPropertyValue("holCode",propData[1])
testCase.setPropertyValue("holYear",propData[2])
testCase.setPropertyValue("holDate",propData[3])
//Rename request test step for readability in the log; append code and year
testCase.getTestStepAt(0).setName("GetHolidayDate-" + propData[1] + "-" + propData[2])

The first (non-comment) line opens our file for reading via a BufferedReader object (if you put the csv file into a location other than C:\Temp, you need to modify this line accordingly).  The object is assigned to a property of the built-in context variable.  This allows us to share the object (and consequently the source file) and keep track of where we are in the file across multiple test steps and iterations.  Remember that the setup script is run prior to any test steps; it's not included in the steps that get looped over, which is what we want-- in particular, we don't want this first line to run more than once.

In the remaining lines of the setup script we read in the first line of our file, assigning it to a local variable firstLine (line 4).  In line 6, the line is split (using the comma to define our split points) and the split data items are assigned to an array called propData.  In lines 7 - 10 the members of the array are assigned to test case properties.  In the final line we're actually renaming the request test step by appending the holiday code and year; this is optional but makes the script log more readable.

Once the test case properties have been read into the script, property expansion is used to generate the test request:

The GetHolidayDate test request using test case properties
After reading in the first line of our data file, for example, the test request is sent using "US" as the countryCode, "NEW_YEARS" as the holidayName, and "2014" as the year.

A single XPath Match assertion is set up to validate our test request.  You can use property expansion with an XPath Match to specify the expected result; in this case, the holDate test case property is used:

Using property expansion with the XPath Match assertion for the GetHolidayDate test request
The next step in the test case is the ReadNextLine Groovy test step, responsible for managing iterations.  Here's the script contained in this test step:

/*Read in the next line of the file
  We can use the same object created in the Setup script because it
  was assigned to the context variable.*/
def nextLine = context.sourceFile.readLine()
/*If the end of the file hasn't been reached (nextLine does NOT equal null)
  split the line and assign new property values, rename test request steps,
  and go back to the first test request step*/
if(nextLine != null){
 def propData = nextLine.split(",")
 def curTestCase = testRunner.testCase
 curTestCase.setPropertyValue("holCountry",propData[0])
 curTestCase.setPropertyValue("holCode",propData[1])
 curTestCase.setPropertyValue("holYear",propData[2])
 curTestCase.setPropertyValue("holDate",propData[3])
 //Rename request test step for readability
 curTestCase.getTestStepAt(0).setName("GetHolidayDate-" + propData[1] + "-" + propData[2])
 testRunner.gotoStep(0)
}

The first non-comment line reads in the next line of our file, assigning it to the nextLine variable.  If we're not yet at the end of the file (if we were, nextLine would equal null), we basically go through the same code we saw before in the setup script: the data in the nextLine variable is split into an array, the values in the array are assigned to our test case properties, and the test request step is renamed to identify the current holiday code and year.  One difference here is that there's no built-in testCase variable as there is in the setup script, so we have to get a reference to the test case object via the built-in testRunner variable and assign it the variable curTestCase.  The final line uses the gotoStep() method of the testRunner variable to send execution back to the first test step in the test case-- the test request step.  After the new iteration of the test request step (with the newly assigned property values), the ReadNextLine step runs again, etc., until we finally do reach the end of the file.  Once this happens, the if statement fails and the gotoStep() method is bypassed.  Execution passes on to the short test case teardown script:

//Cleanup: rename test step to its original name and close the file reader
testCase.getTestStepAt(0).setName("GetHolidayDate")
context.sourceFile.close()

In this script we're just renaming the request step back to its default name (no identifying values appended) and the source file is closed.

Try running the test suite-- the resulting test suite log for the GetHolidayDate test case should look something like this (if necessary, click the TestSuite Log button at the bottom of the test case dialog to display the log):

We can see right away that something failed for the GetHolidayDate test request-- and because we were modifying the test step names as we went, we can see it occurred for Emmeline Pankhurst Day, the third row in the source file.  If you want to see more information about the failure, you can simply click on the error to launch the Message Viewer.  Here you can see the request and response XML (including the values resulting from property expansions) to get a better idea of where the problem may be.  In this case, our source file indicated that the expected date for the holiday was 2010-04-17, but the service returned 2010-07-14 (this was in fact an intentional error in the source file).
That illustrates one way to perform data looping with the soapUI free version, but by no means the only way; keep in mind that soapUI Pro includes additional tools to facilitate data driven testing.  If data driven testing is something you intend to use heavily or your particular case requires a complex implementation, the additional cost of the paid version of soapUI may be worth the savings in time and effort invested in maintaining scripts.

XPath in soapUI Part 3: Common XPath Functions

XPath supports quite a few functions that can be used to make your soapUI assertions even more sophisticated, including functions for different data types, aggregate functions to perform calculations across multiple values, etc.  In this post I'll look at a few basic functions with some examples; for a more extensive list I recommend this page at w3schools.com.  The examples reference the same sample project used in the last few posts, targeting a web service that returns data about holidays; you can download the project here if you haven't done so already.   The post also assumes you're familiar with namespaces, namespace prefixes, and basic XPath syntax, covered in previous posts.

Count() and Numeric Functions

The count() function (not too surprisingly) counts the number of matches found for a given expression.  For example, the following XPath expression counts the number of Holiday elements returned in the GetHolidaysAvailable test response; we could use it in an assertion to confirm that 24 holidays are returned:

count(//Holidays)

There are several numeric functions similar to the count() function that perform operations on expressions that return multiple matches.  For example, Holidays elements returned by the GetHolidaysAvailable test request have a numeric rowOrder attribute.  We can use the following functions to find their minimum, maximum, average, and sum:

min(//Holidays/@msdata:rowOrder)   (returns 0.0)
max(//Holidays/@msdata:rowOrder)   (returns 23.0)
avg(//Holidays/@msdata:rowOrder)   (returns 11.5)
sum(//Holidays/@msdata:rowOrder)   (returns 276.0)

String Functions

There are quite a few XPath functions for working with string data.  The contains(), starts-with(), and ends-with() functions each take two strings as their arguments; the first string (typically the result of an XPath expression) is tested accordingly for the presence of the second string.  For example, run the following XPath assertion against the GetHolidaysAvailable response:

contains(//Holidays[6]/Name , 'Patrick')   (returns true)

The XPath expression //Holidays[6]/Name evaluates to "St. Patrick's Day" (the Name of the sixth Holidays element in the response).  As the first argument to the contains() function, the string "St. Patrick's Day" is checked to see if it contains the second argument, the string "Patrick".  Consequently, the function returns the Boolean true.

Some example expressions using the starts-with() and ends-with() functions:

starts-with(//Holidays[8]/Key , 'GOOD_')   (returns true)
ends-with(//Holidays[13]/Name , 'Mayo')   (returns true)

Date Functions

XPath also provides some functions for working with dates, including functions to extract individual date/time components (months, years, minutes, etc.) from date and time values.  Here are examples we could use with the GetHolidaysForYear test request to confirm the month and day of Cinco de Mayo:

month-from-dateTime(//Holidays[Name = 'Cinco de Mayo']/Date)   (returns 5)
day-from-dateTime(//Holidays[Name = 'Cinco de Mayo']/Date)   (returns 5)

There are corresponding functions to retrieve the year, hours, minutes, or seconds from a full date and time value-- year-from-dateTime(), minutes-from-dateTime(), etc.-- plus functions to do the same with date- or time-only values-- year-from-date(), month-from-date(), hours-from-time(), seconds-from-time(), and so on.

Not() Function

In the last post, we saw that it's possible to use the logical operators and and or with XPath expressions.  The not() logical function takes an expression that results in a Boolean value and negates it, returning true if the original expression evaluated to false or false if the original expression evaluated to true. Not() is frequently used with the count() function to check that a given condition doesn't exist in test response XML.

For example, the GetHolidaysForYear test request returns holidays for a specified year.  One check we might want to do is to confirm that all of the dates are indeed within the specified year.  We know that no matter how many holidays are returned, zero holidays should be returned that don't fall in the specified year (we'll use 2013 in this case).  Here's a function that uses count(), not(), starts-with(), and logical operators to confirm that at least one holiday is returned (to make sure we're not just counting zero elements to begin with-- which would indicate something else is going wrong) and none of them fall outside 2013:

count(//Holidays) > 0 and count(//Holidays[not(starts-with(Date , '2013'))]) = 0   (returns true)

The first half of the assertion (before the and operator) should be pretty self-explanatory, but let's analyze the second part from the inside out, beginning with the starts-with() function, which checks the value of the Date element (as a string) to see if it begins with "2013".  The result of the starts-with function, a Boolean value, is negated by the not() function.  All of this is used as a predicate (within brackets) to target certain Holidays elements-- i.e., return only those Holidays elements with Date child elements that don't start with "2013".  Finally, matches found for that expression are counted by the count() function.  As we expected, the result of the count() function is 0-- none of our Holidays elements have dates that don't fall in 2013.

The example illustrates one of the potential pitfalls when working with longer, more complex expressions and multiple functions-- navigating the tangle of brackets and parentheses.  If you receive an "Invalid XPath expression" error trying to evaluate a complex expression like this, be sure to double-check the order and count (always in opening and closing pairs) of parentheses and brackets.  Also, if you get stuck building an expression with nested functions, you may find it helpful to back up and try working from the inside out, if possible, checking to make sure each part works by itself before putting all the pieces together.

In the next post, we'll look at using XPath with property transfers in soapUI.

XPath in soapUI Part 2: More Complex XPath Expressions

In the last post we looked at a simple example of an XPath assertion in soapUI-- in fact, we could have used a basic Contains assertion to accomplish pretty much the same thing.  In this post, we'll see a little more of the power of XPath and work on some more complex expressions.

Using the same sample project we used in the last post, let's look at the GetHolidaysForYear test case and test request.  Run it and look at the response XML:

An excerpt from the GetHolidaysForYear response XML
Where the GetHolidayDate request we looked at in the last post essentially returned one significant node, there are multiple nodes in which we could be interested here; consequently, there are some extra layers of complexity to deal with in our XPath expression.  Let's say we want to confirm the correct date is reflected for Valentine's Day in 2013, for example.  There are multiple Holidays elements; our XPath expression has to identify which one corresponds to Valentine's Day and check the value of its Date child element.

Create a new XPath assertion and click the Declare button to have soapUI automatically generate namespace declarations.  You may notice an odd thing here: one of our declared namespaces (for prefix "ns1") is an empty string:

Namespace declarations auto-generated by soapUI
If you look again at the response XML, you'll see there is a namespace defined (without a prefix) as an empty string in the NewDataSet opening tag.  This appears to be a negation of an earlier namespace definition in the GetHolidaysForYearResponse tag-- without this empty string namespace defined for the NewDataSet element, it would inherit the namespace defined for the GetHolidaysForYearResponse element ("http://www.27seconds.com/Holidays/").  The upshot: for the NewDataSet element (and any children without their own prefixes) we actually don't have to use any namespace prefix in our XPath expressions.

Now we have to figure out how to specify the Holidays element in our response that corresponds to Valentine's Day.  There are a few ways of doing this.  One pretty straightforward way is through the use of an index, placed within brackets after the Holidays element in our expression.  Note that with XPath, index numbers are 1-based as opposed to 0-based-- the first instance has index 1, the next has index 2, etc.  Valentine's Day is the 4th holiday reflected in the response, so we can construct our XPath assertion like this:

//Holidays[4]/Date

As you'll remember from the last post, "//Holidays" signifies that we're looking for a Holidays element anywhere (at any level in the tree) in the response XML.  The "[4]" following the name of the Holidays element references the fourth instance of the element occurring in the response.  Finally, "/Date" indicates that we're looking for the value of a Date element-- but note the use of a single-slash here instead of a double-slash: this limits our results to a Date element that's a child of the element before the slash.  The expression is equivalent to saying, "Return the value of the Date element that's a direct child of the fourth Holidays element."  Click the "Select from current" button to evaluate the expression, and it should return the expected date of "2013-02-14T00:00:00-05:00".

Of course, if an operation doesn't always return its data in the same order, using an index may not be sufficient.  You can use other types of expressions within the brackets to identify a particular occurrence of an element.  For example:

//Holidays[Name = "Valentine's Day"]/Date

This XPath expression isn't much different from the previous one (in fact, it evaluates to the same value), but instead of an index it uses a comparison operation within the brackets.  This expression finds the instance of a Holidays element that has a Name child element with the value "Valentine's Day", and then returns the value of the Date element for that instance.

You can also work with XML attributes, data specified within a tag as opposed to between matching tags.  For example, each Holidays element has a unique attribute called rowOrder that can be used to identify a particular instance.  We can use the following expression to get the Date value for Cinco de Mayo:

//Holidays[@msdata:rowOrder = "10"]/Date

Just as above, we're using the expression in the brackets to identify a particular instance of a Holidays element.  The "@" symbol indicates that we're looking for an attribute and not a standard element.  Note that we have to be careful with namespace prefixes here-- unlike the Holidays and Date elements, the rowOrder attribute has its own namespace prefix in the response XML and therefore requires a corresponding prefix ("msdata") in our XPath expression.

In addition to the "=" (equals) operator, XPath supports all the comparison operators you might expect: "!=" (not equals), ">" (greater than), "<" (less than), ">=" (greater than or equal to), and "<=" (less than or equal to).  Additionally, you can use these comparison operators in conjunction with "and" and "or"-- some examples:

//Holidays[@diffgr:id = "Holidays17" and @msdata:rowOrder <= 20]/Key   (evaluates to "LABOR")

//Holidays[@msdata:rowOrder >= 100 or @diffgr:id = "Holidays21"]/Name   (evaluates to "Thanksgiving")

In the next post we'll look at some common XPath functions.