|
SPECweb99 Frequently Asked Questions |
SPECweb99 is a software benchmark product developed by the Standard Performance Evaluation Corporation (SPEC), a non-profit group of computer vendors, systems integrators, universities, research organizations, publishers and consultants. It is designed to measure a system's ability to act as a web server for static and dynamic pages.
SPECweb99 is the successor to SPECweb96, and continues the tradition of giving Web users the most objective, most representative benchmark for measuring web server performance. SPECweb99 disclosures are governed by an extensive set of run rules to ensure fairness of results.
The benchmark runs a multi-threaded HTTP load generator on a number of driving "client" systems that will do static and dynamic GETs of a variety of pages from, and also do POSTs to, the SUT (System Under Test).
SPECweb99 provides the source code for an HTTP 1.0/1.1 load generator that will make random selections from a predetermined distribution. The benchmark defines a particular set of files to be used as the static files that will be obtained by GETs from the server, thus defining a particular benchmark workload.
The benchmark does not provide any of the web server software. That is left up to the tester. Any web server software that supports HTTP 1.0 and/or HTTP 1.1 can be used. However, it should be noted that variations in implementations may lead to differences in observed performance.
To make a run of the benchmark, the tester must first set up one or more networks connecting a number of the driving "clients" to the server under test. The benchmark code is distributed to each of the drivers and the necessary fileset is created for the server. Then a test control file is configured for the specific test conditions and the benchmark is invoked with that control file.
SPECweb99 is a generalized test, but it does make a good effort at stressing
the most basic functions of a web server in a manner that has been standardized
so that cross-comparisons are meaningful across similar test configurations.
This is a minor release which fixes several issues found in the client test harness after the release. There are code changes to the module HTTP/HT.c and manager only. These changes include:
Minor documentation edits have been made to clarify the benchmark's operation in light of the changes shown above as well as to make a few corrective edits that were missed in the initial document review. User's should review these modified sections:
SPEC requires that all SPECweb99 published after Nov 1, 1999 use the new Release 1.01. All licensees of SPECweb99 will be provided with an update kit which will include the source for HTTP/HT.c and all updated documentation.
SPECweb99 Release 1.02 is a minor release which fixes several issues found in the client test harness after release 1.01. There are code changes to the files abstract.c, configure.in, random.c, manager, and client.c. These changes include:
Documentation Changes:
SPEC requires that all SPECweb99 results submitted after September 18, 2000
use the new Release 1.02. All licensees of SPECweb99 will be provided with
an update kit which will include all updated source and documentation.
SPECweb99 measures the maximum number of simultaneous connections, requestingthe
predefined benchmark workload, that a web server is able to support
while still meeting specific throughput and error rate requirements. The connections
are made and sustained at a specified maximum bit rate with a maximum segment
size intended to more realistically model conditions that will be seen on the
Internet during the lifetime of this benchmark.
SPECweb99 includes many major enhancements:
The SPECweb99 workload simulates the accesses to a web service provider, where the server supports the home page for a number of different organizations. Each home page is a collection of files ranging in size from small icons to large documents or images. As in the real world, certain files within the home page are more popular than others. The dynamic GETs simulate the common practice of "rotating" advertisements on a web page. The POSTs simulate entry of user data into a log file on the server, such as might happen during a user registration sequence.
In this release, SPECweb99 has added dynamic services for which real-world data was available. Dynamic services are still developing too rapidly at this point to allow all of the latest features to be included. Dynamic content, consisting of 16% POSTs, 41.5% GETs, and 42% GETs with cookies, and 0.5% CGI GETs, comprises 30% of the overall workload.
The workload is based on analysis of server logs from a variety of popular Internet servers and some smaller Web sites. The workload defines four classes of files which are accessed. These have the following file sizes: less than 1 KB, 1 to 10 KB, 10 to 100 KB, and 100 KB to 1 MB. There are nine files in each class, with sizes distributed evenly through the range for that class.
The access patterns to the files were determined from the analysis of the web server logs. SPECweb99 directs 35% of its activity to the smallest class, 50% to the 1-to-10-KB class, 14% percent to the 10-to-100-KB class, and 1% to the largest files. Within each class there are non-linear distributions of accesses, reflecting the fact that certain files are more popular than others.
Recent studies have shown that file accesses on a server follow a Zipf distribution
(see, for example, http://www.useit.com/alertbox/zipf.html).
Thus, SPECweb99 directory and within-class accesses are generated using the
Zipf distribution function. The resulting overall distribution is very close
to actual measured distributions on real servers.
Yes. SPEC is providing a six month transition period from the date of the
SPECweb99 announcement (June 14, 1999). During this period, SPEC will accept,
review and publish results from both benchmark versions. After this period,
results from SPECweb96 will no longer be accepted by SPEC for publication.
No. Although the benchmarks are similar, they cannot be compared, since SPECweb99 uses an entirely different metric than SPECweb96. SPECweb99 also has different file access distributions and a mix of different types of server queries. The dynamic part of the SPECweb99 workload has no SPECweb96 equivalent, so there is no way to make meaningful comparisons between the two.
Initial SPECweb99 results are available on SPEC's Web site: http://www.spec.org/web99/. Subsequent results will be posted on an ongoing basis following each two-week review cycle: results submitted by the semi-weekly deadline are reviewed by web committee members for conformance to the run rules, and if approved at the end of that period are then publicly released.
SPECweb99 is a standardized test. The SPEC membership - leading vendors, systems integrators, universities, research organizations, publishers and consultants - has agreed on a single benchmark with one standardized implementation and workload.
Before any SPECweb99 results are published, SPEC requires that the system under test and the methodology used adhere to agreed-upon standards (the run rules). All results available through SPEC include full disclosure information that reveals exactly what configurations have been used to obtain a particular result.
SPECweb99 results published on SPEC's web site provide standardized, comparable results for those who cannot run their own web server performance tests.
SPECweb99 is a standardized benchmark, which means that it is an abstraction of the real world. For example, SPECweb99 does not attempt to model latency associated with obtaining data across a wide-area network (WAN) such as the Internet. This kind of behavior is difficult to simulate at this stage, since it requires elaborate hardware and software.
SPECweb99 is not the right tool for sizing a server. SPECweb99 was not designed as a capacity planning tool. However, it does provide information on how web servers handle this specific workload. The workload uncovers several key components of a good web server, including LAN performance, processing power, and memory bandwidth, to name a few.
SPECweb99 is scheduled for public release in June 1999. It will be available on CD-ROM for $800. A discount is available for universities and other non-profits as well as for SPECweb96 licensees. To order, contact SPEC's administrative office.
The benchmark comes with the code necessary to build and run the driver system(s), as well as sample API modules for the dynamic content and the code for building the dataset that will be requested from the web server. It is up to the tester to provide the web server and to install the dataset created by the SPEC tools.
SPECweb99 does not provide web server software, but any web server that is HTTP/1.0 or 1.1 compliant can be used. In addition to licensed server software, public domain server software can be used to run SPECweb99.
The SPECweb99 CD-ROM contains:
First, of course, you'll need properly running web server software on your server. On at least one client system, you'll need a full ANSI C programming environment to build the benchmark from the provided source code. Binaries will be provided for some system architectures.
Web Server: computer(s) with enough stable storage for fileset and web server logs
Client(s): computer(s) with enough stable storage for client software
Network: any TCPIP compatible network connecting the web server and clients
There may have been some issues that have been raised about the benchmark since it was released. We are keeping a SPECweb99 issues repository.
If your issue is not amongst the known issues, then bring it to the attention of SPEC.
Only SPECweb99 licensees can submit results. SPEC member companies submit results free of charge. Non-members may submit results for an additional fee. All results are subject to a two-week review by web committee members. Non-member submissions are also subject to a preliminary review. If they pass preliminary review, they may be submitted for the standard member review, and if approved will be published by SPEC upon payment of a fee. First-time submitters should contact SPEC's administrative office.
SPECweb99 submissions must include both the raw output file and the source code for the dynamic implementation used. Submitters may also be required to submit the last 20% of the web server log file upon request.
The current version of the run rules can be found at http://www.spec.org/web99/docs/runrules.html.
The SPECweb99 white paper contains detailed design information on the benchmark and workload. as well as instructions for installing and running the benchmark. See: http://www.spec.org/web99 for the white paper and other available information on SPECweb99.
Revised: Fri Aug 11 10:40:02 EDT 2000