Notes on SPECweb96 Suite

Releases

SPEC sees Web benchmarking, like all of its benchmarking efforts, as an ongoing effort. Thus, we expect to have several releases of these benchmarks as the benchmarks and the web itself evolves.

We are looking to cut our first CDs in a matter of weeks. We do have a working code base, and are in the stages of finishing documentation and resolving the smaller implementation issues.

We do admit that our first release is not likely to be the last word in WWW benchmarking, thus we already are expecting to start work on the second release (covering areas such as CGI, security, etc.) as soon as we can ship the first release.

Workload

For first release, we are focusing the workload on GETs. We'll be testing POSTs and other aspects of Web serving in the second release.

The workload is built on a model of a (hypothetical) web provider. This web provider offers people space for them each to place their web pages so that they can be accessed. Thus, each "member" of this web provider has their own web space on the server, and a number of different pages within that space. Each benchmark client then simulates a number of web browsers by referencing random pages throughout the web provider's server.

The file sizes and access frequencies have been developed by looking at log files from NCSA, HP, HAL Computers, and even a Comics Web Site. We then reviewed that data in light of access pattern information from CommerceNet, and Netscape.

Design

SPECweb is largely leveraged from SPEC's other networking benchmark SFS; we used the existing test scaffold and have made the necessary adaptations to drive HTTP protocol rather than NFS protocol.

The benchmark scaffold supports multiple drivers exercising a System Under Test (SUT). The multiple drivers are controlled by a "prime client" which coordinates timings and amount of load generated as well as the collection and calculation of results.

Each driver has logic for the generation, submission, and validation of HTTP requests. Each driver spends its time in a loop of: select a random URL, submit that request, validate, do it again. The request, the timings, and any return status are recorded.

Participation

SPECweb is a collaborative effort. Some of the more active participants include:

• CommerceNet
• Digital
• HAL Computer Systems
• Hewlett-Packard
• IBM
• Intel
• OpenMarket
• Netscape
• Siemens Nixdorf
• Silicon Graphics
• Spyglass
• Sun Microsystems