JMeter: A Data-Driven Test Script - Part 2

This is the second post looking at a sample JMeter script; in the first we covered the basic structure of the script.  In this post we'll take a look at the request samplers and associated config and assertion elements.  Here's a look at our script tree again:


The methods of our service are divided into two groups.  The three element specific requests are grouped together beneath a Loop Controller (labelled "ElementRequests Loop") with a CSV Data config element to facilitate testing the requests using different inputs.  Each of the three element specific requests are further nested under Simple Controllers-- these are purely structural, giving each sampler its own "branch" in the tree so we can group them with their own assertions and config elements.

For the sake of comparison, the first service method (GetElementSymbol) uses a WebService (SOAP) request sampler:


In the WSDL helper section you can enter the web service's WSDL URL and click the Load WSDL button to let JMeter try to identify available services; selecting a service and clicking the Configure button automatically populates key fields like Protocol, SOAPAction, etc. (to see this behavior, try temporarily adding a new sampler in the Workbench section, entering "http://www.webservicex.net/PeriodicTable.asmx?WSDL" for the URL).  Notice that the Server Name and Path fields are blank here-- these were already specified in the HTTP Request Defaults config element near the top of our script tree; since the sampler is at a lower level, these defaults are applied to it.  This sampler also illustrates that we can use variables in component name fields ("GetElementSymbol - SOAP Request - ${ElName}")-- useful for making our results more readable.  The "Read SOAP Response" checkbox must be enabled if you intend to use response data with a listener (which we are doing here).

We still have to provide the actual body of the request-- here's the GetElementSymbol request XML:


As the documentation for the WebService (SOAP) request sampler notes (and the dialog title screams in capital letters), it's been deprecated (you can read a discussion re: deprecation of the sampler here).  The recommended technique is to make SOAP requests via the HTTP sampler.  The remaining three requests are performed with the HTTP Request sampler; here's the GetAtomicNumber sampler:


By default, an HTTP Request sampler is configured to pass a series of parameters; to use the HTTP sampler for a SOAP request, you have to switch to use the Post Body tab in the HTTP Request section and put your request there.  Additionally, the HTTP sampler is missing a few of the fields available in the SOAP request sampler (the SOAPAction field, for example) so we have to add these to our HTTP Requests using a Request Header config element with each HTTP Request.  Here's the Request Header associated with the GetAtomicNumber sampler:


Each sampler is also grouped with a Response Assertion element to check response data for expected results:


As you can see, variables can be used in assertions, which makes our data-driven technique possible: each row of our CSV data file contains an input variable (an element name passed in as a request parameter) along with variables for expected response content.

The Pattern Matching Rules section of the Response Assertion dialog warrants some clarification.  The Contains and Matches options are used when your patterns to test are regular expressions (the Matches option means the entire response needs to match your regular expression).  The Equals and Substring options are used when your patterns to test should be evaluated as literal (case-sensitive) strings (the Equals option means the entire response needs to match the provided string).  The Not checkbox essentially reverses the test-- i.e., the assertion passes if the pattern is not found or matched.

JMeter: A Data-Driven Test Script - Part 1

In this post and the next I'm going to break down a JMeter script that illustrates a simple data-driven test technique and some of JMeter's components.  The first part will focus on the structure of the script, covering some key logic and configuration controllers; since they warrant a little more attention, the second part will look at the script's request samplers and closely associated components. The script (JMeter script files have a JMX extension) and csv data file used by the script are available for download here in a single zip file.  After unzipping the files, open the JMX file in JMeter (File - Open in the file menu).  By default, the csv file is expected to be in a folder called C:\Temp\PerTableData, but you can put it anywhere you'd like as long as you modify the CSV Data Controller accordingly.

Here's what the script tree should look like once you've opened it:


To see the script in action, click on the node for the View Results Tree listener (where it's easiest to see the script's progress), then click on the Play button at the top of JMeter's main window (the button with the large green arrow).  The script runs against a web service that provides data about the periodic table of elements.  There are four methods provided by the service, which are split into two groups in the script-- three of the methods are element specific (they return data based on an individual element provided as a parameter) and the last simply returns a list of all the elements in the periodic table (if you've read my post on data driven testing in soapUI, this should all look very familiar-- this script runs against the same service and is similarly structured).

Since this is strictly a functional test script, we only need a single user configured in the Test User element at the top of the script:


The HTTP Request Defaults config element at the top of the script specifies a default server, path, etc., to use for request samplers.  Since it's placed at the top of the tree, it's applied to all requests; however, you can override these default values by specifying different values locally (within an individual request's own configuration dialog).  One potential "gotcha" to note: the server name field in this element does not take a full URL-- the "http://" prefix should be omitted when specifying the server.


The "Duration LT 5 Sec" Duration Assertion asserts that all requests should complete within 5000 milliseconds (5 seconds).  As with the HTTP defaults element, its placement at the topmost level of the script tree means it's applied to all requests.


The "ElementRequests Loop" Loop Controller establishes a "branch" to the tree that loops indefinitely (an If Controller placed within the loop will terminate it based on our source data):


Within our loop, the first element is a CSV Data Set Config element:


The element specifies the location of our source CSV file to use to read in variable values-- if you put this file somewhere other than the default location, you'll have to change the Filename entry here.  It also specifies the variable names we're going to use in our script-- each variable name corresponds to a column in our CSV file-- so the first item in each row is assigned to the ElName variable, the second item is assigned to the ElNum variable, etc.  Note that we don't have to do any additional work to manage reading this file-- no maintaining a separate counter to keep our place, for example-- each time it's encountered in our loop, JMeter reads in the next line.  Although we're not using it in this case (we only have one user configured), note that you can "share" data from the CSV file between multiple threads/virtual users via the "Sharing mode" options-- meaning each time any thread encounters the CSV file in its execution, it reads the next line.  So you could have one thread read in data from row 1, the next thread read in data from row 2, etc.

To prevent the loop from running indefinitely, our test requests are nested beneath an If Controller that checks to see if the end of our CSV file has been reached.


When JMeter attempts to read in data from our CSV file after it's already read in the last row, "<EOF>" is returned.  That serves as a flag that it's time to exit the loop-- or more specifically, when "<EOF>" is not returned as the value of the ElName variable, that serves as a flag that it's okay to continue to execute our requests.  The condition statement illustrates how to reference variables: enclose the name in braces and precede it with "$" (remember that our variable names were configured in the CSV Config element).  Note that in our condition statement ("${ElName}"!="<EOF>") the name of our string variable is enclosed in quotes-- this syntax may seem unusual to some; see JMeter's component reference for some more condition statement examples.

Skipping the request samplers and related config elements for now (again, we'll look at those in the next post), the script contains two listener components which collect and report run data.  The Results Tree listener shows request and response data in detail-- even if you don't intend to use this listener in your finalized script, it's good for troubleshooting when you're building a script, allowing you to see exactly what's sent and received and create your assertions accordingly.


In the next post we'll look specifically at the script's request samplers and compare the deprecated SOAP Request sampler with the HTTP Request sampler.

JMeter: An Overview

Apache JMeter is an open source testing tool that's primarily designed for load/performance testing (a key feature is the ability to simulate multiple concurrent users), but it's useful for functional testing as well.  Although I'm going to look at it primarily in the capacity of a web services testing tool a la soapUI, it's capabilities don't end there-- according to its web site, testable resources include "files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more."


The main JMeter window
JMeter's UI features a tree view (on the left in the screenshot above) where you can easily build and visualize your script.  Right-click on a given node in the tree to add child elements or perform other actions (these vary by component type).  You can also drag and drop nodes (including their child nodes) to quickly re-structure your script, moving them before or after other elements or making them children of other nodes.

JMeter's many many script components are divided into several categories.  Check out the extensive component reference for details on all of them.  Some of the key ones:

Threads (Users): Thread elements define and configure the virtual user(s) making requests in your JMeter script.  At least one thread is required before samplers can be added.

Samplers: Test requests are configured using sampler elements, which include samplers for HTTP requests, FTP requests, SMTP requests, and SOAP requests (note, however, that the SOAP request sampler has been deprecated-- the HTTP Request element can be used for SOAP requests as well).

Logic Controllers: Logic controllers add loops and conditional execution of test requests, making it possible to create some fairly sophisticated scripts in terms of flow control.  These include a Loop Controller, ForEach Controller, If Controller, While Controller, etc.

Config Elements: Config elements are used for a variety of configuration-related tasks, including setting/managing variables and creating default parameters to use in request samplers.  The "scope" of some config elements is determined by their level in the script tree.  For example, settings in an HTTP Request Defaults config element are applied to applicable samplers at or below its tree level, allowing you to use a single HTTP Request Defaults element to define the endpoint and path for multiple HTTP Request samplers (saving you the trouble of having to re-enter the same information for each sampler).

Assertions: JMeter Assertions are used to verify expected results, including assertions for expected response time, content, size, etc.  As with config elements, assertions can be applied to multiple samplers by placing them at the same tree level (or higher) in a script.  In the screenshot above, for example, the "Duration LT 5 Sec" assertion is applied to all the samplers in the script since they all fall beneath it in the tree hierarchy.

Listeners: You can add listeners to your script to record, consolidate, and display results in a variety of ways-- listeners include a table view of results, a tree view of results, assertion results summaries, and graphs.

In my next post I'll look at a sample script that should help illustrate some of these components.