Version 1.08
Last modified: November 30, 2006
Section 2 - Installing SPECjAppServer2004
Section 3 - Building and Deploying SPECjAppServer2004
Section 4 - Running SPECjAppServer2004
SPECjAppServer2004 is a J2EETM benchmark to measure the scalability and performance of J2EE servers and containers and is based on the J2EE1.3 Specification. SPECjAppServer2004 strives to stress the middle-tier rather than the client tier or the database server tier.
This document is a practical guide for setting up and running a SPECjAppServer2004 benchmark. This user guide discusses some, but not all, of the rules and restrictions pertaining to SPECjAppServer2004. Before continuing with the benchmark, we strongly recommend you read the complete SPECjAppServer2004 Run and Reporting Rules contained in the kit. For an overview of the benchmark architecture, see the SPECjAppServer2004 Design Document also contained in the kit.
"SPECjAppServer2004 JOPS" is the SPECjAppServer2004 performance metric and denotes the jAppServer Operations Per Second completed during the Measurement Interval. "SPECjAppServer2004 JOPS" is composed of the total number of business transactions completed in the Dealer Domain, added to the total number of workorders completed in the Manufacturing Domain, normalized per second.
EAStress2004 is a research workload created on the basis of SPECjAppServer2004 benchmark. It is only provided as a tool for research use such as testing and experimentation in situation where the benchmark run rules cannot be met due to technical reasons (e. g., usage of projects under development, usage of experimental platforms). EAStress2004 results are not reviewed by SPEC. EAStress2004 results can in no way be compared with SPECjAppServer2004 results.
"EAStress2004 HASOPM" is the EAStress performance metric and denotes the Hundreds of Application Server Operations Per Minute completed during the Measurement Interval. "EAStress2004 HASOPM" is composed of the total number of business transactions completed in the Dealer Domain, added to the total number of workorders completed in the Manufacturing Domain, normalized to hundreds per minute.
The Domain is a logical entity that describes a distinct business sphere of operations. The four SPECjAppServer2004 domains are: Dealer, Manufacturing, Supplier and Corporate.
The Driver refers to the client code that drives the benchmark, monitors requirements, and reports results.
SPECjAppServer2004 Kit refers to the complete kit provided for SPECjAppServer2004. This includes the SPECjAppServer2004 Reference Beans, the Driver, load programs, and documentation.
SPECjAppServer2004 Reference Beans refers to the implementation of the Enterprise Beans provided for the SPECjAppServer2004 workload.
The Supplier Emulator is a Java Servlet that can run inside any Java enabled web server, and emulates the process of sending and receiving orders to/from suppliers.
The SUT is the System UnderTest and is comprised of all components that are being tested. This includes Application Servers/Containers, Database Servers, network connections, etc. The Driver and Supplier Emulator are not part of the SUT.
The SPECjAppServer2004 workload emulates an automobile manufacturing company and its associated dealerships. Dealers interact with the system using web browsers (simulated by the driver) while the actual manufacturing process is accomplished via RMI (also driven by the driver). This workload stresses the ability of Web and EJB containers to handle the complexities of memory management, connection pooling, passivation / activation, caching, etc. The SPECjAppServer2004 Design Document includes a complete description of the workload and the application environment in which it is run. This section of the user's guide describes software and hardware environment required to run the workload.
Although SPECjAppServer2004 can be run on a single machine for testing purposes, compliance with the SPECjAppServer2004 Run and Reporting Rules requires that the driver and supplier emulator be run on a machine outside the SUT. Therefore, a compliant hardware configuration must include a network and a minimum of two systems – one or more systems to run the components within the SUT and at least one system to run the driver and supplier emulator outside the SUT. A typical configuration is illustrated below.
SPECjAppServer2004 is a J2EE application that requires a J2EE 1.3 compatible application server as well as a Relational Database Management System (RDBMS) to run as part of the SUT. Outside the SUT, a Java Runtime Environment (JRE) version 1.3 or later is required for the driver and supplier emulator. A web server supporting HTTP 1.1 is also required for the supplier emulator
The SPECjAppServer2004 Kit is supplied as either a tar, gzip, or zip file that should be extracted into a directory. We will call this top level directory ${JAS_HOME}.
SPECjAppServer2004 can be run on any Application Server that is compliant with the J2EE 1.3 Specification and has passed the Certification Test Suite (CTS) for that specification. Most of the code is implemented in Java, however there are a few scripts used to start certain programs. These scripts are provided in .sh format for Unix platforms and in .bat format for Windows.
ant : The ant installation directory.
bin : Driver and database load scripts. All additional scripts except specific database related scripts will be in this directory.
classes : This is a directory where compiled Java classes are placed. It comes with all the necessary class files necessary to run SPECjAppServer2004. Re-compilation of the source files is unnecessary to run the application.
classes/META-INF : Temporary place for holding deployment descriptors just before packaging into JAR format files.
classes/WEB-INF : Temporary place for holding web deployment descriptors just before packaging the WAR files.
config : Environment configuration and properties files used by the Driver, make and load programs. Once SPECjAppServer2004 has been successfully deployed, this is the only place where the configuration should be modified.
config/security : Security policy files used for the benchmark programs. These should not be modified.
docs : The documentation for all SPECjAppServer2004 beans generated by running 'make doc'.
jars : The JAR files created by the make files. The files distributed in this directory are driver.jar, launcher.jar, charts.jar and xerces.jar. The application server specific jars for deployment need to be generated by the user.
License : Contains license document files for the product
output : Default directory in which output of runs go.
reporter : Contains the source for the reporter which consumes results from a run to create the submission package for SPEC.
schema : Schema information and static data to be loaded into the database.
schema/sql : Generic SQL scripts for creating the tables, etc. in the database. These should be copied and modified as necessary for a particular database product.
src : Root directory of the source files. This contains generic and application server specific make files.
src/com : Base for the com.sun.jini.debug and com.sun.jini.thread package source.
src/deploy : The deployment descriptors.
src/docroot : HTML and JSP pages used for the web interface.
src/org : Entry into the org.spec.jappserver package source.
Note: For some of the popular DBMS products, DBMS specific files are in schema/<DBMS vendor> and DBMS specific SQL scripts are in schema/<DBMS vendor>/sql.
All SPECjAppServer2004 Java classes are located in the org.spec.jappserver package. The following lists the sub-packages of org.spec.jappserver :
The beans adhere to the EJB 2.0 specification. This package only provides beans that are deployable as Container Managed Persistence (CMP) entities.
Of the packages listed above, corp,orders,mfg, and supplier contain the beans used for the four SPECjAppServer2004 domains respectively; the util package is included once with each of the other domains, more on this later. Each domain package contains distinct sub-packages for each EJB and its helper classes. The EJB package contains a sub-package named "ejb" which holds the bean implementation including the home and remote interfaces. For instance, the following shows the fully qualified class names for the OrderLineEnt entity bean:
org.spec.jappserver.orders.orderlineent.ejb.OrderLineEntHome
org.spec.jappserver.orders.orderlineent.ejb.OrderLineEnt
org.spec.jappserver.orders.orderlineent.ejb.OrderLineEntPK
org.spec.jappserver.orders.orderlineent.ejb.OrderLineCmpEJB
The helper classes for the orders domain are in the helper sub-package under orders as follows:
org.spec.jappserver.orders.helper.CustomerInfo
org.spec.jappserver.orders.helper.CustomerStatus
org.spec.jappserver.orders.helper.InsufficientCreditException
org.spec.jappserver.orders.helper.ItemQuantity
org.spec.jappserver.orders.helper.OrderStatus
Besides the four domains, several other packages are included:
The common package contains the source code for helper classes used by all the beans (i.e. it is common across different domains as well as beans).
The util package contains the beans used to generate primary keys for the various domains.
The supplier.emulator package contains the Supplier Emulator Servlet.
The driver package contains the SPECjAppServer2004 drivers.
The load package contains the database data generation and load programs required to load the SPECjAppServer2004 database(s) according to the benchmark scaling rules.
xerces.jar is used by Delivery Servlet in Supplier domain and the Supplier Emulator Servlet. Add this jar file in the Application server classpath if XML DOM Parser packages are not part of your application server. To include this jar in the Tomcat server, copy xerces.jar file to ${TOMCAT_HOME}/lib directory.
There are several components required to build, deploy, test and run SPECjAppServer2004. These are :
Building and deploying the benchmark requires Ant 1.6.2, which is provided as part of the SPECjAppServer2004 kit.
Several steps must be accomplished prior to running the SPECjAppServer2004 benchmark:
For the Standard Workload, you can create a single database that houses all the 4 Domains. For the Distributed Workload, you must create 4 separate databases, one for each domain.
Standard SQL scripts for creating the database schema are provided in schema/sql. These are intended to give a starting point for creating schemas for other database products. It is beyond the scope of this user guide to provide guidance for the installation and configuration of the various RDBMs available on the market. For convenience database creation scripts are provided for Oracle 9i, DB2 and Sybase in schema/<DBMS>. These are example scripts provided by the RDBMs vendors and can be modified, but serve as a starting place for deployment. However, see section 2.4 of the SPECjAppServer2004 Run and Reporting Rules for restrictions on modifications to the schema scripts.
Loading of the database's is deferred until the environment file and associated batch / script files have been updated in subsequent steps.
This section describes the steps required to build the SPECjAppServer2004 ear in preparation for deployment. These instructions assume that the target application server has been installed and the environment variable ${J2EE_HOME} points to the directory above the lib directory that contains the j2ee.jar. This is generally the home of the AppServer being tested. Please refer to your specific application server's installation instructions for information about a specific product.
In the ${JAS_HOME}/config directory there are files named appserver and default.env which contain the environment variables used to deploy and run the benchmark.
The file $JAS_HOME/config/appserver is used to locate the correct env file for deployment and execution. The contents of this file are appended with .env and the resulting file becomes the properties file used. You may either update the supplied default.env file or make a copy of it. For example, if you copy default.env to myAppServer.env update the file $JAS_HOME/config/appserver to contain myAppServer.
Edit the ${JAS_HOME}/config/default.env properties file (or your new copy).
You need to set the following environment properties:
| Kit Specific Settings | |
| JAS_HOME | Specifies the root directory
for the SPECjAppServer2004 kit. For example, if deploying
the benchmark on Windows in C:\working\SPECjAppServer2004
update the environment variable like: JAS_HOME=C:/working/SPECjAppServer2004 |
| FS | This variable defines the File Separator for the deployment / driving system. For users deploying on Windows you must update the ${FS} environment variable from ':' to ';' otherwise you will get "Class not Found" Exceptions when running the scripts provided with this benchmark. |
| JAVA_HOME | Normally the Application Server
being tested provides a JRE to run their application server.
If you choose to use the supplied one update this environment
variable to root directory for the JRE. Otherwise, you can
point to the JRE of your choice. Note:The JDK pointed to here is the one used to drive workload. This is independent of the JRE used to start the Launcher which is started in the driver.sh or driver.bat scripts depending on your driving system. |
| J2EE_HOME | The Application Server being tested should provide a library (e.g. j2ee.jar) containing its implementation of the J2EE specification. This directory specifies the directory above the lib directory. |
| JAVAX_JAR | The path to the Application
Server's library (e.g. j2ee.jar) containing its implementation
of the J2EE specification. Note: In versions of the kit up to 1.03, the JAVAX_JAR property was not included in the sample properties file ${JAS_HOME}/config/default.env. Make sure you don't forget to set this property in your env file. |
| BIND_WAIT | Specifies the number of seconds that the Launcher should wait after starting all processes to allow them to initialize and connect to the controller. Default is 5 seconds if it is not specified. |
| CLASSPATH related Entries | |
| Note: The Launcher (invoked in driver.sh / driver.bat ) prepends the file ${JAS_HOME}/jars/driver.jar to the beginning of the classpath so that the driver cannot be overridden. | |
| CLASSPATH | Contains the various jar files required by the driver. Note: The corp.jar, mfg.jar, orders.jar, and supplier.jar are assumed to contain the stubs for the AppServer being tested. In most cases these are the jars that are created during the deployment process for the AppServer being tested. Review the default.env file for an example of this variable. |
| JDBC_CLASSPATH | Provides the path and jar files needed to access the database; this will be unique for each RDBMS. This path is referred to in the load.sh or load.bat script file as appropriate for your operating environment |
| SUT AppServer Information | |
| Specify the name of the host where the Emulator will find the DTD for the POs and for the Audit Beans to locate and reset logs. | |
| JAS_HOST | Identifies the system(s) that house the DeliveryServlet as well as specifying the DTD that the Emulator will need to read to validate the PO XML sent as part of Supplier interaction with the Emulator. |
| JAS_PORT | The port number of the WebContainer / HTTP entry point to the SUT. Default is port 80. |
| JAS_PREFIX | Set this to prepend a prefix for the Web URL. This is not necessary in most deployments.Provided to improve flexibility for testing. |
| Emulator Information | |
| Like the similar variables used for the SUT these variables provide information to the SUT on how to access the Emulator as well as for the auditing Beans to determine the number of emulator interactions. These parameters are used during deployment as well as at runtime. | |
| EMULATOR_HOST | Specifies the name of the system that is acting as the emulator. |
| EMULATOR_PORT | Default port for HTTP requests made to the Emulator. |
| EMULATOR_PREFIX | Used to prepend a prefix for the Emulator URL. This is not necessary in most deployments but is provided for flexibility in testing and research. |
| RMI Settings | |
| These variables are used during deployment and runtime by the driver. They are Application Server specific. Refer to the documentation provided by your Application Server vendor for specific information. Appendix A provides the settings provided by some Application Server vendors for these values. | |
| NAMING_PROVIDER | Identifies the URL of the Remote RMI endpoint to provide name resolution and to send RMI requests. Consult the documentation for the tested AppServer for the correct setting. |
| EMULATOR_PORT | Default port for HTTP requests made to the Emulator. |
| JNDI_CLASS | This setting (like the NAMING_PROVIDER) is unique to the AppServer. Each AppServer should provide a class name to effect JNDI lookups. |
| Driver JVM Settings | |
| These settings are used for the launched components of the driver. | |
| JVM_OPTIONS | Identifies the URL of the Remote RMI endpoint to provide name resolution and to send RMI requests. Consult the documentation for the tested AppServer for the correct setting. |
| JAVA | Used by the Launcher to start the various subprocesses for the Orders Domain (HTTP) as well as the Manufacturing Domain (RMI) |
The ${JAS_HOME}/bin directory contains
scripts to setup the shell environment for subsequent steps. There are
two files provided; a .sh for Unix environments and a .bat file
for Windows. Below is a sample .sh file. Update these variables
to the values as specified in the env file above and then execute
the shell script. For example: ../setenv.sh
| #!/bin/sh # JAS_HOME=/SPECjAppServer2004 JAVA_HOME=/usr/java/jdk1.4.1 J2EE_HOME=/yourAppServer CONFIG_DIR=${JAS_HOME}/config APPSSERVER=default ENVFILE=${CONFIG_DIR}/${APPSSERVER}.env echo "The following environment settings are in effect for SPECjAppServer2004" echo \* ========================= \* echo JAVA_HOME=${JAVA_HOME} echo JAS_HOME=${JAS_HOME} echo CONFIG_DIR=${CONFIG_DIR} echo APPSSERVER=${APPSSERVER} echo \* ========================= \* |
During subsequent build steps the classes of the SPECjAppServer kit will be combined with the set of J2EE deployment descriptors used to create a deployable ejbjar file as well as combine the various EJB jars into a deployable unit called an Enterprise ARchive (EAR). The various Deployment Descriptors (DDs) used in the kit need to have some information in them updated with some of the information provided in the environment file. As a result, during the build process the ANT utility will look in the directory ${JAS_HOME}/src/deploy/<appserver> to find the appropriate deployment descriptors.
In the ${JAS_HOME}/src/deploy/ directory create a directory which follows the value you specified in step 3.1.2.1 and copy the reference DDs into this new directory. If the file ${JAS_HOME}/config/appserver file had been updated to myAppServer the following commands on UNIX would need to be executed:
$ cd ${JAS_HOME}/src/deploy
$ mkdir myAppServer
$ cp reference/* myAppServer
For Windows users:
C> cd %JAS_HOME%\src\deploy
C> mkdir myAppServer
C> copy reference\* myAppServer
Application Server vendors may provide additional DDs required for their product to complete the deployment process. They should also provide a set of instructions that work within the context of this build environment to complete the steps for their products.
The following commands for both Unix and Windows can be used to build the EJB Jar, Web Application aRchive (WAR)and Enterprise ARchive (EAR) files. We assume that the file ${JAS_HOME}/config/appserver has been updated to myAppServer in step 3.1.2.1.
Unix |
Windows |
|---|---|
| $ cd ${JAS_HOME} | C> cd %JAS_HOME% |
| $ ant/bin/ant clean | C> ant\bin\ant clean |
| $ ant/bin/ant -Dappserver=myAppServer | C> ant\bin\ant -Dappserver=myAppServer |
If any errors occur at this step it is generally a setup / directory mis-match problem. Please review all settings carefully prior to contacting SPEC for support.
There are several files in the $JAS_HOME/config/ directory named *db.properties. These files are used by the load utility programs to find and connect to the database server you will be using for your tests. You will need to know the correct URL for your JDBC driver; please refer to your JDBC driver's documentation for format information. You will also need to know the user and password to connect to the database.
dbUser = spec
dbPassword = spec
dbURL = jdbc:<yourDBURL>:specdb
dbDriver = <your.database.package.dbDriver>
jdbcVersion = 2
Note: In case you are using different database username/password than the default spec/spec you will also have to edit the dbUser and dbPassword properties in the above files.
The properties files provided in the $JAS_HOME/config/ directory contain some URLs provided by some DB vendors.
Now load the database:
$ cd $JAS_HOME
$ ant/bin/ant loaddb
Note: To load the database for higher Injection Rate runs use the bin/loaddb.sh script.
Usage:
loaddb <scale> [<numThreads> <-doFlatFiles>]
Where:
<scale> required,
is the injection rate at which the benchmark will run
<numThreads> is optional, if not specified,
the default is 1
<-doFlatFiles> is an optional flag, and if
not specified the loader doesn't generate flat files.
When -doFlatFiles option is specified, the loader will generate the files as follows
${pipeDir}/datafiles/Ir_YYY/tableName/tableName_XXXX
YYY in Ir_YYY represents
the injection rate specifed
XXXX in tableName_XXXX represents
a four digit number starting from 0000 and could have as many as the
number of threads specified.
pipeDir is a configurable property and by default
is set to /tmp
For Example:
/tmp/datafiles/Ir_10/c_customer/c_customer_0000
/tmp/datafiles/Ir_10/c_customer/c_customer_0001
Data written to these files are separated by a delimiter. By default it is "|" and could be configured using the property loadDelimiter.
Database vendor specific load scripts could be used to load these files into the database.
Note: On Windows it is recommended to increase the heap size allocated to the loader by editing the bin/loaddb.bat script. This may speed up the loading process significantly.
To build and deploy emulator on Tomcat server, edit the $JAS_HOME/config/tomcat.env properties file.
$ cd $JAS_HOME
$ ant/bin/ant -f tomcat.xml
$ cp jars/Emulator.war $TOMCAT_HOME/webapps
$ rm -rf $TOMCAT_HOME/webapps/Emulator
The remaining steps to deploy the SPECjAppServer.ear file created in the ${JAS_HOME}/jars/ directory depend on the Application Server you are using and whether or not you are using extended DDs provided by them. Each Application Server is unique and it is impossible here to capture the exact instructions to complete the deployment. Here are a list of steps that are required to complete the deployment if not using vendor supplied DDs. Please refer to Application Server documentation for help in accomplishing these steps:
Running SPECjAppServer2004 requires that the driver be configured for your environment and particular application server. Configuring the driver properly requires understanding how the driver works, which is described below.
The SPECjAppServer2004 Driver consists of several Java programs and is designed to run on multiple client systems, using an arbitrary number of JVMs to ensure that the Driver has no inherent scalability limitations. Note that none of the client systems are part of the SUT.
The Master machine: This is the machine from which you drive the benchmark run. It is typically one of the client systems.
The Agent machines: These are the client systems on which the various workloads run. The Master machine can also be an Agent machine.
The Driver consists of the following components:
The Driver communicates with all the Agents using RMI. The Driver reads the run properties and configures the Agents appropriately. Each Agent will then run as many threads of their respective workload. For example, the DealerAgent will run DealerEntry. The number of threads is determined by the scaling rules of the specification and are equally distributed amongst all Agents. Each thread runs independently, executing the workload according to the rules defined in the spec. When the run completes, the Driver co-ordinates with the Agents to retrieve all the stats and prints out the reports.
In the past, some people have had problems with the way the Driver gets its InitialContext and lookups. This section explains exactly how this is done.
The Driver obtains the JNDI InititalContext by executing the following code in each of the client threads in Agent processes:
try {
context = new InitialContext();
} catch (NamingException ne) {
throw new RemoteException(InitialContext failed. " +
ne);
}
Each of the bean home interfaces that the applications require is looked up using the absolute home name of the bean. An example is shown below:
OrderSesHome orderSesHome =
(OrderSesHome) PortableRemoteObject.narrow(
context.lookup("OrderSesHome"), OrderSesHome.class);
If your app server needs extra properties to be set for the InitialContext, this can be done by setting them in jndi.properties in the CLASSPATH or changing the variable JAVA in config/<appserver>.env. The example below shows a generic implementation. Specific values for several application servers are provided in the subsequent sections.
JAVA="$JAVA_HOME/bin/java
-Djava.naming.factory.initial=<vendor naming factory
goes here>
-Djava.naming.provider.url=<vendor url goes here>"
This step assumes you have already completed the steps in 3.1.2.1. Make sure that the config directory on the driving system is synchronized with the updates made earlier. In addition to those changes you must also:
Now, run the driver by running the script bin/driver.sh [<driver host>] from the $JAS_HOME directory.
You can abort a run at any time by pressing Ctrl-C. This will cause all the processes to exit gracefully.
If there were errors during pervious run, M_largeorder table may have entries which will affect the "SPECjAppServer2004 JOPS" of the run. So make sure before each run the M_largeorder table is empty by reloading the database or by deleting entries from M_largeorder table using a SQL command.
If no <driver host> is specified the default local host is used.
We have sometimes experienced binding exceptions at the start of the first time run. For now, just cancel the script (by pressing Ctrl-C) and rerun it again. This problem usually does not persist to the second run.
The driver.sh script starts all the Agents on the Master machine. If you want to start the Agents on different client systems, the script needs to be edited appropriately. "bin/driver.sh <driver host>" must be started on the other client systems after starting the script on the Master machine. <driver host> is the name of the Master machine. The properties file must be configured correctly on the client systems, with the appropriate configurations in the $JAS_HOME/config/run.properties for the OrderAgent and the MfgAgent to be run on the client systems. In the run.properties file the runOrders= and runMfg= properties actually specify the number of Orders / Manufacturing agents to run on the target system.
A driver.bat script is provided for running SPECjAppServer2004 on MS Windows systems.
After completing a run, the SPECjAppServer2004 driver will generate a result.props file in the output directory containing the raw results from the benchmark. The contents of result.props are used in the Submission File from which the SPECjAppServer2004 report is generated.
The Submission File contains a detailed description of the SUT in text format that is used by the SPECjAppServer2004 reporter to produce a report suitable for publication.
To create a SPECjAppServer2004 Submission File:
SPECjAppServer2004 includes a utility for generating a results page called the reporter. The report contains a summary of the key results and a complete system description in a format designed to be consistent with other SPEC reporting pages.
To run the reporter, use the following command:
$ java -classpath reporter.jar reporter [-a] <submission file>.txt
Where <submission file>.txt is the Submission File created in the previous step.
An HTML file named <filename>.report.html is created by default.
The "-a" option will create a text report page named <filename>.report.txt.
Once you have a successful run, you can submit the results to the SPEC OSG Java subcommittee for review. To submit a result to SPEC:
Every submission goes through a minimum two-week review process, starting on a scheduled SPEC OSG Java sub-committee conference call. During the review, members of the committee may ask for additional information or clarification of the submission. Once the result has been reviewed and accepted by the committee, it is displayed on the SPEC web site at http://www.spec.org/.
Copyright (c) 2004 Standard Performance Evaluation Corporation