Standard Performance Evaluation Corporation

With corporate Web sites getting millions of hits per day, the choice of Web-server software can be critical. Web servers that don't respond crisply under heavy loads can bog down network connections, deny service for visitors, and cause network failures. · For administrators of large sites or others seeking to differentiate among the wide variety of Web-server software packages, the Standard Performance Evaluation Corp.'s SPECweb96 benchmark can in many cases help determine which Web-server software performs best on a particular set of hardware systems and network connections. · Though SPEC officially released SPECweb96 1.0 in September, production problems with the Windows NT version delayed release of the CD-ROM software until the end of December. This software can evaluate the performance of Web-server software running on virtually any Unix system or Windows NT platform.· Like SPEC's other benchmarks, SPECweb96 is a standardized performance benchmark created in a spirit of cooperation among competitors. As
a nonprofit organization, SPEC oversees the development and creation of benchmarks by using the technical resources of member companies. The result is a benchmark that has been examined by all interested parties and is considered a fair test of performance by them.
For SPECweb96, companies as diverse as Digital Equipment, Hewlett-Packard, IBM, Intel, Netscape Communications, Siemens Nixdorf, Silicon Graphics, Sun Microsystems, and others helped shape the benchmark.

How It Works
A SPECweb96 test bed consists of a server machine that runs the Web-server software to be tested and a set number of client machines. The client machines use the SPECweb96 software to generate a workload that stresses the server software. The workload is gradually increased until the server software is saturated with hits and the response time degrades significantly.
The point at which the server is saturated is the maximum number of HTTP operations per second that the Web-server software can sustain. That maximum number of HTTP operations per second is the SPECweb96 performance metric that is reported. For example, the SPECweb96 performance metric of the sample machine in the table and graph on page 58 is 626 HTTP operations per second.
Although SPECweb96 can do an admirable job of testing Web-server performance, you may not want to rush out and pay $800 for the CD-ROM without first considering what SPECweb96 is testing and what kind of test bed you'll need. It can take some serious network resources to properly test Web-server software, and your site may not be ready to take on such software configuration responsibility. Also, as with most benchmarks, you'll need close-to-laboratory conditions to make accurate comparisons among systems.
That means you can't expect to test the server over a WAN such as the Internet, since it is difficult to account for all possible network bottlenecks that might prevent saturation of the Web-server software. To get accurate results, you need a relatively quiet LAN between your test clients and server system.
How many client systems will you need to saturate your Web-server software with hits? That answer depends on your configuration. For example, in results reported to SPEC, Digital used two AlphaServer 4000 clients-each with four 400-MHz CPUs and 256 Mbytes of RAM-to saturate the Zeus Web-server software from Zeus Technology in England, running on an AlphaServer 4000 with a single 400-MHz CPU.
To sustain the throughput necessary to saturate the Web server, the company used a 100-Mbps FDDI LAN. This configuration produced an impressive SPECweb96 performance metric of 1,157 HTTP operations per second.
On the other hand, to saturate Zeus Web-server software running on an HP 9000 with a single 132-MHz PA-RISC CPU, HP needed only three HP 9000 clients, each with a single 160-MHz CPU and 128 Mbytes of RAM. This configuration produced a SPECweb96 performance metric of 426 HTTP operations per second. In other words, you need enough client horsepower to saturate the server system.
At the same time, you don't want the LAN to be a performance bottleneck. If the network becomes saturated before the server system does, you won't be able to adequately test the server.
You can determine the amount of network throughput you'll need by taking a guess at your saturation mark and then inserting it into this formula (saturation point * 6/1,000) to determine megabits per second. For the HP machine mentioned above, the company needed a LAN that could sustain 2.6 Mbps (426 * 6/1,000). The SPECweb96 documentation has details on the derivation of this formula.

Limitations
A benchmark is just a benchmark. It doesn't tell you everything you want to know about the tested systems, and sometimes it doesn't tell you much at all. Fortunately, SPEC is very clear about the limitations of the SPECweb96 benchmark.
Perhaps the biggest limitation is the nature of the workload. In the real world, Web servers are hit with a multitude of different workloads. Some Web-server software may spend most of its time handling CGI (Common Gateway Interface) requests. Other Web-server software may be handling dynamic multimedia requests.
The workload for SPECweb96 was determined by looking at log files for major sites such as NCSA, HP, CommerceNet, and Netscape. The SPEC committee then came up with a set of file sizes, access frequencies, and access patterns that it felt accurately reflected the traffic of a hypothetical Web provider that offers space for people to put up home pages.
The SPECweb96 clients simulate a number of Web browsers referencing pages throughout the Web site. The SPECweb96 benchmark uses only HTTP GET commands (the command used to retrieve a Web page), but does not do POST commands or CGI calls, or test any security mechanisms.
Does this sound like the Web traffic your site gets? If not, SPECweb96 may not be the best performance test for your site.
SPECweb96 uses only the HTTP 1.0 protocol. It does not support any of the HTTP 1.1 mechanisms, such as Keep-Alive, which can dramatically improve Web-server response time by keeping the network connection open instead of creating a new connection for each Web-page component downloaded.
One of the best things-and one of the worst-about the SPEC benchmarks is that they are distributed in source code form. This is great because you can inspect the benchmark to see what it's doing or find out exactly why it's not working. But it also means you need an ANSI C programming environment to build the test suite before installing it on your Web server and client systems.


Other Compilers
The Windows NT version of SPECweb96 was built using Microsoft Visual C++ 4.0, but could probably be adapted to other compilers. The Unix version of SPECweb96 includes Make files for about 16 Unix variants, including Linux, UnixWare, Solaris, HP/UX, and several System V and BSD variants.
SPECweb96 uses Perl language scripts both to administer the test and report results. Fortunately, a Perl 5.0 interpreter is included on the CD-ROM.
Obviously, you'll also need some Web-server software. Because all Web servers use the HTTP 1.0 protocol, any Web server can be used.
Depending on the kind of traffic your site gets, SPECweb96 may be a useful performance comparison test for your Web-server software. For many sites, it's a useful performance metric, since HTTP GET commands comprise the vast majority of Web-site traffic.

What's Next?
If your traffic doesn't fit the SPECweb-96 mold, however, you'll be pleased to hear that the SPEC folks aren't sitting still. The next generation of Web-server benchmark, SPECweb97, is already under way and will support HTTP 1.1 as well as POST commands, CGI calls, and more interesting workloads.
Accurate performance testing of network server software has always been a tricky business, and no single benchmark can compare every facet of performance. With SPEC's backing, experience, and rigorous reporting requirements, SPECweb-96 is your best bet for Web-server software performance comparisons.
More information about SPECweb96, including approved testing results, can be found at SPEC's Web site, http://www.spec.org.