|
SPECweb99_SSL Frequently Asked Questions |
SPECweb99_SSL is a software benchmark product developed by the Standard Performance Evaluation Corporation (SPEC), a non-profit group of computer vendors, systems integrators, universities, research organizations, publishers and consultants. It is designed to measure a system's ability to act as a secure web server for static and dynamic pages.
SPECweb99_SSL is a software benchmark product designed to test secure web server performance using HTTP over the Secure Sockets Layer Protocol (HTTPS). The benchmark is built upon the SPECweb99 test harness and uses the same workload and file set (see: http://www.spec.org/osg/web99/).
SPECweb99_SSL continues the tradition of giving Web users the most objective, most representative benchmark for measuring secure web server performance. SPECweb99_SSL disclosures are governed by an extensive set of run rules to ensure fairness of results.
The benchmark runs a multi-threaded HTTPS load generator on a number of driving "client" systems that will do static and dynamic GETs of a variety of pages from, and also do POSTs to, the SUT (System Under Test) over SSL.
SPECweb99_SSL provides the source code for an HTTP 1.0/1.1 over SSL (Secure Socket Layer Protocol) load generator that will make random selections from a predetermined distribution. The benchmark defines a particular set of files to be used as the static files that will be obtained by GETs from the server, thus defining a particular benchmark workload.
The benchmark does not provide any of the web server software. That is left up to the tester. Any web server software that supports HTTP 1.0 and/or HTTP 1.1 over SSL (HTTPS) can be used. However, it should be noted that variations in implementations may lead to differences in observed performance.
To make a run of the benchmark, the tester must first set up one or more networks connecting a number of the driving "clients" to the server under test. The benchmark code is distributed to each of the drivers and the necessary fileset is created for the server. Then a test control file is configured for the specific test conditions and the benchmark is invoked with that control file.
SPECweb99_SSL is a generalized test, but it does make a good effort at stressing
the most basic functions of a secure web server in a manner that has been standardized
so that cross-comparisons are meaningful across similar test configurations.
SPECweb99_SSL measures the maximum number of simultaneous connections, requesting
the predefined benchmark workload, that a secure web server is able
to support while still meeting specific throughput and error rate requirements.
The connections are made and sustained at a specified maximum bit rate with
a maximum segment size intended to more realistically model conditions that
will be seen on the Internet during the lifetime of this benchmark.
SPECweb99_SSL includes:
SPECweb99_SSL adds SSL support to the existing SPECweb99 workload. In SPECweb99_SSL, the client systems will generate the same mix of static and dynamic GETs and POSTs as SPECweb99, however now the clients will negotiate SSL connections with the server and all requests and responses will be encrypted. Each connection will establish an SSL session and the session may be resumed for several sequential HTTP operations.
The non-SSL portion of the SPECweb99_SSL workload is identical to the SPECweb99 workload and simulates the accesses to a web service provider, where the server supports the home page for a number of different organizations. Each home page is a collection of files ranging in size from small icons to large documents or images. As in the real world, certain files within the home page are more popular than others. The dynamic GETs simulate the common practice of "rotating" advertisements on a web page. The POSTs simulate entry of user data into a log file on the server, such as might happen during a user registration sequence. The dynamic content, consisting of 16% POSTs, 41.5% GETs, and 42% GETs with cookies, and 0.5% CGI GETs, comprises 30% of the overall workload.
The workload is based on analysis of server logs from a variety of popular Internet servers and some smaller Web sites. The workload defines four classes of files which are accessed. These have the following file sizes: less than 1 KB, 1 to 10 KB, 10 to 100 KB, and 100 KB to 1 MB. There are nine files in each class, with sizes distributed evenly through the range for that class.
The access patterns to the files were determined from the analysis of the web server logs. The benchmark directs 35% of its activity to the smallest class, 50% to the 1-to-10-KB class, 14% percent to the 10-to-100-KB class, and 1% to the largest files. Within each class there are non-linear distributions of accesses, reflecting the fact that certain files are more popular than others.
Studies have shown that file accesses on a server follow a Zipf distribution
(see, for example, http://www.useit.com/alertbox/zipf.html).
Thus, SPECweb99 directory and within-class accesses are generated using the
Zipf distribution function. The resulting overall distribution is very close
to actual measured distributions on real servers.
No. SPECweb99 is still available for measuring the performance of HTTP web
servers that do not use SSL. SPEC will accept, review and publish results
from both benchmarks.
No. Although the benchmarks have similarities, results from one cannot be
compared to the other, since SPECweb99_SSL adds SSL encryption to the SPECweb99
workload. However for the same system, the differences between
the SPECweb99 and SPECweb99_SSL results could be used to help evaluate
the performance impact of SSL encryption on the server. Public
comparisions of SPECweb99 and SPECweb99_SSL results would be considered
a violation of the run and reporting rules.
Initial SPECweb99_SSL results are available on SPEC's Web site. Subsequent results will be posted on an ongoing basis following each two-week review cycle: results submitted by the semi-weekly deadline are reviewed by web committee members for conformance to the run rules, and if accepted at the end of that period are then publicly released.
SPECweb99_SSL is a standardized test. The SPEC membership - leading vendors, systems integrators, universities, research organizations, publishers and consultants - has agreed on a single benchmark with one standardized implementation and workload.
Before any SPECweb99_SSL results are published, SPEC requires that the system under test and the methodology used adhere to agreed-upon standards (the run rules). All results available through SPEC include full disclosure information that reveals exactly what configurations have been used to obtain a particular result.
SPECweb99_SSL results published on SPEC's web site provide standardized, comparable results for those who cannot run their own web server performance tests.
SPECweb99_SSL is a standardized benchmark, which means that it is an abstraction of the real world. For example, SPECweb99_SSL does not attempt to model latency associated with obtaining data across a wide-area network (WAN) such as the Internet. This kind of behavior is difficult to simulate at this stage, since it requires elaborate hardware and software.
SPECweb99_SSL is not the right tool for sizing a server. SPECweb99_SSL was not designed as a capacity planning tool. However, it does provide information on how web servers handle this specific workload. The workload uncovers several key components of a good secure web server, including LAN performance, processing power, and memory bandwidth, to name a few.
SPECweb99_SSL is scheduled for public release in March 2002. It will be available on CD-ROM for $800. A discount is available for universities and other non-profits as well as for SPECweb99 licensees. To order, contact SPEC's administrative office.
The benchmark comes with the code necessary to build and run the driver system(s), as well as sample API modules for the dynamic content and the code for building the dataset that will be requested from the web server. It is up to the tester to provide the web server and to install the dataset created by the SPEC tools.
SPECweb99_SLL does not provide secure web server software, but any secure web server that is HTTP/1.0 or 1.1 compliant and supports SSLv3 can be used. In addition to licensed server software, public domain server software can be used to run SPECweb99.
The SPECweb99_SSL CD-ROM contains:
First, of course, you'll need properly running secure web server software on your server. On at least one client system, you'll need a full ANSI C programming environment to build the benchmark from the provided source code. Binaries will be provided for some system architectures.
Secure Web Server: computer(s) with enough stable storage for fileset and web server logs
Client(s): computer(s) with enough stable storage for client software
Network: any TCP/IP compatible network connecting the web server and clients
There may have been some issues that have been raised about the benchmark since it was released. We are keeping a SPECweb99_SSL issues repository.
If your issue is not amongst the known issues, then bring it to the attention of SPEC.
Only SPECweb99_SSL licensees can submit results. SPEC member companies submit results free of charge. Non-members may submit results for an additional fee. All results are subject to a two-week review by web committee members. Non-member submissions are also subject to a preliminary review. If they pass preliminary review, they may be submitted for the standard member review, and if approved will be published by SPEC upon payment of a fee. First-time submitters should contact SPEC's administrative office.
SPECweb99_SSL submissions must include both the raw output file and the source code for the dynamic implementation used. Submitters may also be required to submit the last 20% of the web server log file upon request.
The current version of the run rules can be found at http://www.spec.org/osg/web99ssl/docs/runrules.html.
The SPECweb99_SSL Design Overview contains design information on the benchmark and workload. The Run and Reporting Rules and the User Guide with instructions for installing and running the benchmark are also available. See: http://www.spec.org/osg/web99ssl for the available information on SPECweb99_SSL.
Revised: March 6, 2002