|
SPECweb2005 Frequently Asked Questions |
SPECweb2005 is a software benchmark product developed by the Standard Performance Evaluation Corporation (SPEC), a non-profit group of computer vendors, system integrators, universities, research organizations, publishers, and consultants. It is designed to measure a system's ability to act as a web server servicing static and dynamic page requests.
SPECweb2005 is the successor of SPECweb99 and SPECweb99_SSL, offering the capabilities of measuring both SSL and non-SSL request/response performance, and continues the tradition of giving Web users the most objective and most representative benchmark for measuring web server performance.
Rather than offering a single benchmark workload that attempts to approximate the breadth of web server workload characteristics found today, SPECweb2005 has chosen a 3-workload benchmark design: banking, ecommerce, and support. Additionally, the change from a concurrent connection-based workload metric to a simultaneous session-based workload metric is intended to offer a more direct correlation between the benchmark workload scores and the number of users a web server can support for a given workload.
SPECweb2005 continues the tradition of giving Web users the most objective, most representative benchmark for measuring secure web server performance. SPECweb2005 disclosures are governed by an extensive set of run rules to ensure fairness of results.
The benchmark clients run the application program that sends HTTP requests to the server and receives HTTP responses from the server. For portability, this application program and the prime client program have been written in Java. Note that as a logical component, one or more load-generating clients may exist on a single physical system.
The benchmark does not provide any of the web server software. That is left up to the tester. Any web server software that supports HTTP 1.1 and SSL (HTTPS) could be used. However, it should be noted that variations in implementations may lead to differences in observed performance.
To make a run of the benchmark, the tester must first set up one or more networks connecting a number of the driving "clients" to the server under test. The benchmark code is distributed to each of the drivers and the necessary fileset is created for the server. Then a test control file is configured for the specific test conditions and the benchmark is invoked with that control file.
SPECweb2005 is a generalized test, but it does make a good effort at stressing
the most basic functions of a web server in a manner that has been standardized
so that cross-comparisons are meaningful across similar test configurations.
Each SPECweb2005 benchmark workload (SPECweb2005_Banking,
SPECweb2005_Ecommerce, and SPECweb2005_Support) measures the maximum
number of simultaneous user sessions that a web server is able to
support while still meeting specific throughput and error rate
requirements. The TCP connections for each user session are made and
sustained at a specified maximum bit rate with a maximum segment size
intended to more realistically model conditions that will be seen on the
Internet during the lifetime of this benchmark.
No. SPECweb2005 consists of three separate, distinct workloads,
each with its own submetric: SPECweb2005_Banking,
SPECweb2005_Ecommerce, and SPECweb2005_Support. While the individual
submetric scores do indicate the total number of simultaneous user
sessions the server can support, the overall SPECweb2005 metric for a
compliant result is the geometric mean of the three submetrics,
normalized to a reference platform score. Accordingly, a SPECweb2005
score of 100, for example, would indicate the same overall performance
as the reference platform. A score of 200 would indicate an overall
performance that is double that of the reference platform, etc. For
more information, see section 3.1 of the Run and Reporting Rules.
SPECweb2005 includes the following improvements over its predecessors:
SPECweb2005 introduces three entirely new workloads: Banking, Ecommerce,
and Support. These were based upon statistics analyzed from web server
logs and observed client-side behavior. For more information on each
of these workloads, please see the SPECweb2005 design documents.
Yes. SPEC will accept, and review SPECweb99 and SPECweb99_SSL results until
September 27, 2005. After that time only SPECweb2005 results will be
accepted.
No. SPECweb2005 is a completely redesigned benchmark with multiple workloads
and different metrics. Public comparisons of SPECweb99 or SPECweb99_SSL
results to SPECweb2005 results would be considered a violation of the
run and reporting rules.
Initial SPECweb2005 results are available on SPEC's web site. Subsequent results will be posted on an ongoing basis following each two-week review cycle: results submitted by the two-week deadline are reviewed by web committee members for conformance to the run rules, and if accepted at the end of that period are then publicly released.
SPECweb2005 is a standardized test. The SPEC membership - leading vendors, systems integrators, universities, research organizations, publishers and consultants - has agreed on a benchmark suite with one standardized implementation and three new workloads.
Before any SPECweb2005 results are published, SPEC requires that the system under test and the methodology used adhere to agreed-upon standards (the run rules). All results available through SPEC include full disclosure information that reveals exactly what configurations have been used to obtain a particular result.
SPECweb2005 results published on SPEC's web site provide standardized, comparable results for those who cannot run their own web server performance tests.
SPECweb2005 is a standardized benchmark, which means that it is an abstraction of the real world. For example, SPECweb2005 does not attempt to model latency associated with obtaining data across a wide-area network (WAN) such as the Internet. This kind of behavior is difficult to simulate at this stage, since it requires elaborate hardware and software.
SPECweb2005 is not the right tool for sizing a server. SPECweb2005 was not designed as a capacity planning tool. However, it does provide information on how web servers handle this specific workload. The workload uncovers several key components of a good secure web server, including LAN performance, processing power, and memory bandwidth, to name a few.
SPECweb2005 is scheduled for public release in June 2005. It can be purchased on CD-ROM from SPEC at $1,200 for new licensees, $600 for upgrades, and $300 for eligible non-profit organizations. To order, contact SPEC's administrative office.
The benchmark comes with the code necessary to run the driver system(s), the server-side file set generation tools, and dynamic content implementations. It is up to the tester to install and configure the web server and testbed.
The benchmark does not include web server software, but any web server that is HTTP/1.0 or 1.1 compliant and supports SSLv3 can be used. In addition to licensed server software, open source software can be used to run SPECweb2005. See the run rules for more details.
The SPECweb2005 CD-ROM contains:
First, of course, you'll need properly running web server software on your server. On at least one client system, you'll need a 1.4 (or above) JVM for running the prime client and client.
There may have been some issues that have been raised about the benchmark since it was released. We are keeping a SPECweb2005 issues repository.
If your issue is not amongst the known issues, then bring it to the attention of SPEC.
Only SPECweb2005 licensees can submit results. SPEC member companies submit results free of charge. Non-members may submit results for an additional fee. All results are subject to a two-week review by web committee members. Non-member submissions are also subject to a preliminary review. If they pass preliminary review, they may be submitted for the standard member review, and barring any issues will be published by SPEC upon payment of a fee. First-time submitters should contact SPEC's administrative office.
SPECweb2005 submissions must include both the raw output file; during the review process, other information may be requested by the subcommittee.
The current version of the run rules can be found here.
The SPECweb2005 Design Document contains design information on the benchmark and workloads. The Run and Reporting Rules and the User Guide with instructions for installing and running the benchmark are also available. See: http://www.spec.org/osg/web2005 for the available information on SPECweb2005.
Updated: July 17, 2006