SPECjEnterprise®2018 Web Profile Run and Reporting Rules

ACID Properties refers to the Atomicity, Consistency, Isolation and Durability (ACID) properties defined in Transaction Property Requirements

A Business Transaction is a unit of work initiated by a User and may involve any combination of REST Interactions and Web Interactions.

The Components refers to Java EE elements such as EJBs, JSPs, WebSockets, REST APIs, JSFs, and Servlets.

The Configuration Diagram is the picture in a common graphics format that depicts the entire configuration (including the SUT and load drivers). The Configuration Diagram is part of a Full Disclosure. See Configuration Diagram

The Concurrent User Load refers to the rate business transactions are injected into the SUT.

The Cycle Time refers to the time elapsed from the first byte sent by the Driver to request a Business Transaction until the first byte sent by the Driver to request the next Business Transaction. The Cycle Time is the sum of the Response Time and Delay Time.

The Delay Time refers to the time elapsed from the last byte received by the Driver to complete a Business Transaction until the first byte sent by the Driver to request the next Business Transaction. The Delay Time is a function of the Response Time and the Injection Rate. For a required Injection Rate, the Delay Time will be smaller for when Response Times are longer. The driver will adjust the delay to ensure that transactions are delivered in a steady rate.

A Database Transaction is a unit of work on the database with full ACID properties. A Database Transaction is initiated by an enterprise bean as part of a Business Transaction.

The Deployment Unit refers to a Java EE Application Server or set of Application Server instances in which the components from the Insurance application are deployed.

The Driver or Insurance Driver refers to the provided client software used to generate the benchmark workload.

Full Disclosure refers to the information that must be provided when a benchmark result is reported. For the SPECjEnterprise®2018 Web Profile benchmark, Full Disclosure includes the Submission File, Full Disclosure Archive ( FDA ), and Configuration Diagram. See Full Disclosure Archive.

The Full Disclosure Archive ( FDA ) is an archive of files that is part of a Full Disclosure. It contains the benchmark run results and configuration information necessary to reproduce the results.

The Full Disclosure Report ( FDR ) is the formatted benchmark result generated from the Submission File. It includes links to the Full Disclosure Archive ( FDA ) and Configuration Diagram.

The Injection Rate (IR) refers to the rate at which Business Transaction requests are injected into the SUT.

The Measurement Interval refers to a steady state period during the execution of the benchmark for which the test submitter is reporting a performance metric.

A node is the hardware system that runs one or more Java EE Application Servers or Database Server instances. For example, a blade enclosure would generally consist of multiple nodes.

Non volatile storage refers a storage device whose contents are preserved when its power is off.

A Persistent Storage Driver is software that handles the connection between the application server and a persistent storage.

A Relational Database Management System (RDBMS) is a database management system (DBMS) that is based on the relational model.

The Resource Manager is the software that manages a database and is the same as a Database Manager.

The Response Time refers to the time elapsed from when the first transaction in the Business Transaction is sent from the Driver to the SUT until the response from the last transaction in the Business Transaction is received by the Driver from the SUT.

A REST Interaction is a JAX-RS request to the REST portion of the Application.

The SPECjEnterprise®2018 Web Profile Application (or just " Application ") refers to the implementation of the Components provided for the SPECjEnterprise®2018 Web Profile workload.

The SPECjEnterprise®2018 Web Profile Kit (or just " Kit ") refers to the complete kit provided for the SPECjEnterprise®2018 Web Profile benchmark. This includes the SPECjEnterprise®2018 Web Profile Application, Driver, load programs, documentation and other supporting files.

"SPECjEnterprise®2018 WebjOPS" is the primary SPECjEnterprise®2018 Web Profile metric and denotes the average number of successful jEnterprise Web Operations Per Second completed in the Insurance domain during the Measurement Interval. "SPECjEnterprise®2018 WebjOPS" is composed of the total number of Business Transactions completed in the Insurance Domain per second.

The Submission File is a text file containing the Full Disclosure information specified in Submission File. The Submission File is part of a Full Disclosure.

The System Under Test (SUT) comprises all components which are being tested. This includes the Java EE Web Profile Application Servers, Database Servers, network switch(s), etc. It does not include the Driver.

The Test Submitter (or just " Submitter ") refers to the organization that is submitting a benchmark result and is responsible for the accuracy of the submission.

A User represents an individual customer to the System Under Test (SUT)

A Web Interaction is an HTTP request to the Web-based portion of the Application.

The throughput of the SPECjEnterprise®2018 Web Profile benchmark is driven by the activity of the Insurance driver. The throughput of the driver is directly related to the chosen Injection Rate (IR). To increase the throughput, the IR needs to be increased. The benchmark also requires a number of rows to be populated in various database tables. The scaling requirements are used to maintain the ratio between the Business Transaction load presented to the SUT, the cardinality of the tables accessed by the Business Transactions, the IR, and the number of users generating the load.

To satisfy the requirements of a wide variety of customers, the SPECjEnterprise®2018 Web Profile SUT consists of a number of databases and Java EE Web Profile Application Servers that can be mapped to nodes as required. The implementation must not, however, take special advantage of the co-location of databases and Java EE Servers, other than the inherent elimination of WAN/LAN traffic.

The application modules may be combined and deployed on a single Application Server instance. This means that the benchmark implementer can choose to run a single Deployment Unit that accesses a single database that contains the tables of all the modules. However, a benchmark implementer is free to separate the modules into their own Deployment Units, with one or more database instances.

For the Insurance workload the Transaction property requirements described in Transaction Property Requirements must be compatible with the Java EE 7 Web Profile or later specification.

All tables must have the properly scaled number of rows as defined by the database population requirements.

The database schema must be derived directly from the reference schema scripts provided in the databases/default directory of the SPECjEnterprise®2018 Web Profile kit. Derived means that the database schema used must match the schema that the scripts describe, not that the scripts must be used to physically generate the database schema. Modifications to the database schema are allowed only as documented below.

Additional database objects may be added to the reference schema and DDL modifications may be made to the reference schema, however all additions and/or modifications must be disclosed along with the specific reason for the addition/modification. The base tables and indexes in the reference schema cannot be deleted. Views are not allowed. The data types of fields can be modified provided they are semantically equivalent to the standard types specified in the scripts.

Comment : Replacing CHAR with VARCHAR would be considered semantically equivalent. Changing the size of a field (for example: increasing the size of a char field from 8 to 10) would not be considered semantically equivalent. Replacing CHAR with INTEGER (for example: zip code) would not be considered semantically equivalent.

Modifications that a customer may make for compatibility with a particular Database Server are allowed. Changes may also be necessary to allow the benchmark to run without the database becoming a bottleneck, subject to approval by SPEC®. Examples of such changes include:

Scripts or any other files for schema generation provided by the vendors are for convenience only. They do not constitute the reference or the baseline script in the databases/default directory. Deviations from the scripts must be disclosed in the submission file even though the vendor-provided scripts or any other files for schema generation were used directly.

In any committed state the primary key values must be unique within each table. For example, in the case of a horizontally partitioned table, primary key values of rows across all partitions must be unique.

The databases must be populated using the supplied load programs prior to the start of each benchmark run. That is, after running the benchmark, the databases must be reloaded, or table data restored from a backup of a previous load state, prior to a subsequent run. Modifications to the load programs are permitted for porting purposes. All such modifications made must be disclosed in the Submission File.

The database must be accessible at all times for external updates and maintain data access transparency.

Data Access Transparency is the property of the system which removes from the application program any knowledge of the location and access mechanisms of partitioned data. An implementation that uses vertical and/or horizontal partitioning must meet the requirements for transparent data access. The system must prevent any data manipulation operation, including external applications, which would result in a violation of the consistency and isolation requirements. An external application must be able to manipulate any set of rows or columns transparently using the specified table names in the schema script.

External updates may be assumed to update the database version columns.

The submitter must run the SPECjEnterprise®2018 Web Profile Application provided in the kit without modification, except as permitted by these run rules.

Changes necessary to configure and deploy the benchmark in a specific configuration are permitted to the Java Enterprise Edition metadata (configuration files), subject to the requirements of this section. Vendor specific metadata (configuration files) are permitted.

Modifications to persistence.xml and orm.xml consistent with these Run Rules are permitted, but must be provided in the FDR and are subject to review for compliance.

Named query annotations may be overridden in orm.xml provided the new query remains an EJBQL query. Overridden EJBQL queries must return the same data set in the same format as the original query. Replacing EJBQL queries with native queries is not allowed. Exceptions to allow native queries are permitted for queries including "select count(a) from …​" and used exclusively during auditing prior to the ramp-up period, provided they are semantically equivalent.

Comment : The intent of the exception to allow native queries is to reduce the time to perform the auditing counts prior to the run start. There should be no impact to the outcome of the audit tests or the reported benchmark results.

The deployment must assume that the database could be modified by external applications.

The Insurance Driver is responsible for driving load to the Insurance service. For a compliant benchmark run, all of the workload must be driven using the https protocol. When the https protocol is configured in the driver, the driver will use the TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 cipher suite.

The Driver is provided as part of the SPECjEnterprise®2018 Web Profile kit.

The class files provided in the specjwebprofile.jar file of the SPECjEnterprise®2018 Web Profile kit must be used as is. No source code recompilation is allowed.

Faban is used as the infrastructure on which the driver is built. The version included in the SPECjEnterprise®2018 Web Profile benchmark kit must be used as is. No source code recompilation is allowed.

The Driver must reside on one or more systems that are not part of the SUT.

Comment : The intent of this requirement is that the communication between the Driver and the SUT be accomplished over the network.

The Insurance Driver communicates with the SUT using HTTPS. The Insurance Driver uses a single URL to establish a connection with the web tier. If more than one Driver system is used all Driver systems must have the same URL for the Insurance Driver.

Comment : The purpose of the identical URL requirement is to ensure that any load balancing is done by the SUT and is transparent to the Driver systems.

As part of the run, the driver checks many statistics and audits that the run has been properly executed. The driver tests the statistics and audit results against the requirements specified in this document and marks each criteria as "PASS" or "FAIL" in the summary reports. A compliant run must not report failure of any criteria. Only results from compliant runs may be submitted for review and published. Non-compliant runs are not allowed to be published.

Pre-configured Driver decisions, based on specific knowledge of SPECjEnterprise®2018 Web Profile and/or the benchmark configuration, are disallowed.

The Driver systems may not perform any processing ordinarily performed by the SUT, as defined in System Under Test (SUT) Requirements. This includes, but is not limited to:

The Driver records all exceptions in the Driver log. Any errors that appear in the Driver log must be explained in the Submission File. The logging level for a compliant run a minimum logging level of INFO.

The Measurement Interval must be preceded by a ramp-up period of at least 10 minutes at the end of which a steady state throughput level must be reached. At the end of the Measurement Interval, the steady state throughput level must be maintained for at least 5 minutes, after which the run can terminate.

The Atomicity, Consistency, Isolation and Durability (ACID) properties of transaction processing systems must be supported by the SUT during the running of this benchmark.

The SUT comprises all components which are being tested. This includes network connections, Java EE Web Profile Application Servers, Database Servers, etc.

To ensure that SPECjEnterprise®2018 Web Profile results are correctly obtained and requirements are met, the driver will make explicit audit checks by calling auditing components on the SUT. These audit checks are necessary and all tests must pass but they are not sufficient to ensure that the benchmark run is in compliance with these run and reporting rules. Passing the audit checks does not guarantee compliance with all of the run and reporting rules.

These tests are designed and intended to run both prior to ramp-up and also to run immediately following the end of the ramp-down period and it is a requirement that no mechanism is used to delay the start or end of the auditing checks. The driver will include audit results with the run results. The table below lists the individual auditing activities the driver performs, the pass/fail criteria, and the specific purpose of a particular audit:

TABLE 3 : Audit List

The primary metric of the SPECjEnterprise®2018 Web Profile benchmark is jEnterprise Web Operations Per Second ("SPECjEnterprise®2018 WebjOPS").

The primary metric for the SPECjEnterprise®2018 Web Profile benchmark is calculated by the number of Insurance Transactions / sec:

All reported "SPECjEnterprise®2018 WebjOPS" must be measured, rather than estimated, and can be rounded to the nearest whole number. For example, if a measurement yielded "123.55 SPECjEnterprise®2018 WebjOPS", this can be rounded to "124 SPECjEnterprise®2018 WebjOPS".

A graph of the transaction throughput versus elapsed time must be reported for Insurance applications for the entire test run. The x-axis represents the elapsed time from the start of the run. The y-axis represents the throughput in Business Transactions. At least 150 different intervals must be used with a maximum interval size of 30 seconds. The start and end of the Measurement Interval must also be reported. An example of such a graph is shown below.

Benchmark specific optimization is not allowed. Any optimization of either the configuration or products used on the SUT must improve performance for a larger class of workloads than that defined by this benchmark and must be supported by the provider. Optimizations must have the vendor’s endorsement, be supported and should be suitable for production use in an environment comparable to the one represented by the benchmark. Optimizations that take advantage of the benchmark’s specific features are forbidden.

An example of an inappropriate optimization is one that requires access to the source code of the application.

Comment : The intent of this section is to encourage optimizations to be done automatically by the products.

All hardware and software used must be possible to order by customers. For any product not already generally released, the Submission File must include a committed general delivery date. That date must be within 3 months of the result’s publication date. However, if Java and/or Java EE Web Profile related licensing issues cause a change in software availability date after publication date, the change will be allowed to be made without penalty, subject to SPEC® review.

The purpose of including general availability requirements is to ensure that the systems and their hardware and software components actually represent real products that solve real business and computational problems. Detailed Guidelines for General Availability are provided in SPEC® Open Systems Group Policy Appendix C.

All products used must be the proposed final versions and not prototypes. When the product is finally released, the product performance must not decrease by more than 5% of the published "SPECjEnterprise®2018 WebjOPS". If the submitter later finds the performance of the released system to be more than 5% lower than that reported for the pre-release system, then the submitter is required to report a corrected test result.

The intent is to test products that customers will use, not prototypes. Beta versions of products can be used, provided that General Availability (GA) of the final product is within 3 months. If a beta version is used, the date reported in the results must be the GA date.

The 5% degradation limit only applies to a difference in performance between the tested product and the GA product. Subsequent GA releases (to fix bugs, etc.) are not subject to this restriction.

In order to publicly disclose SPECjEnterprise®2018 results, the submitter must adhere to these fair use and reporting rules in addition to having followed the run rules described in this document.

SPECjEnterprise®2018 Web Profile Results must be reviewed and accepted by SPEC® prior to public disclosure.

Any public use of results from this benchmark must also follow the SPEC® Fair Use Rules and those specific to this benchmark.

SPEC® encourages use of the SPECjEnterprise®2018 Web Profile benchmark in academic and research environments. The researcher is responsible for compliance with the terms of any underlying licenses (Java EE Web Profile Application Server, Database Server, hardware, etc.).

The Submission File is a text file containing the information specified in this section.

An example of a Submission File is included in the kit, see "reporter/sample/webprofile/output.submission.properties". The Submission File is generated by the Reporter using the Submission Template and the results of a compliant run. An example of the Submission Template is included in the kit, see "reporter/sample/jEnterprise.webprofile.properties".

The Results Section of the compliant run included in the Submission File must not be modified.

The Full Disclosure Archive (FDA) must be in ZIP format.

A Configuration Diagram of the entire configuration (including the SUT and load drivers) must be provided in PNG, JPEG or GIF format. The diagram should include, but is not limited to:

The various isolation levels are described in ANSI SQL and Java SE documentation for java.sql.Connection. ANSI SQL defines isolation levels in terms of phenomena (P1, P2, P3) that are or are not possible under a specific isolation level. The interpretations of P1, P2, P3 and PD used in this benchmark are as follows:

An isolation level of Strict READ_COMMITTED is defined to disallow P1 and PD but allow P2 and P3.

An isolation level of Strict REPEATABLE_READ is defined to disallow P1, PD and P2 but allow P3.

The ANSI SQL definition for REPEATABLE_READ disallows P2. Disallowing P2 also disallows the anomaly known as Read Skew. Read Skew arises in situations such as the following:

If P2 is disallowed, transaction T2 upon trying to modify X, would either be blocked until T1 completes OR it would be allowed to proceed but transaction T1 would eventually have to be rolled back.

Disallowing P2 also disallows the anomaly known as Write Skew. Write Skew arises in situations such as the following:

If P2 is disallowed, either T1 or T2 would have to be rolled back.

Product and service names mentioned herein may be the trademarks of their respective owners.

Copyright © 2001-2018 Standard Performance Evaluation Corporation
All Rights Reserved