SPEC SFS97 Q&A

---

Answers to Frequently Asked Questions About System File Server (SFS) Release 2.0

Q.) What is SPEC SFS 2.0 and how does this benchmark compare to other network file system (NFS) benchmarks?

A.) SPEC SFS 2.0 is the latest version of the Standard Performance Evaluation Corp.'s benchmark that measures NFS file server throughput and response time. It differs from other NFS benchmarks in that it provides a standardized method for comparing performance across different vendor platforms. The benchmark was written to be client-independent and vendor-neutral. Results undergo peer review before publication on SPEC's public Web site and in its hardcopy newsletter.

Q.) Does this benchmark replace the SPEC SFS 1.1 suite?

A.) Yes. Now that SPEC SFS 2.0 is available, SFS 1.1 licenses are no longer being sold. SPEC is providing a six-month transition period from the date of the SFS 2.0 announcement (December 19, 1997). During this period, SPEC will accept, review and publish results from both benchmark versions. After this period, results from SFS 1.1 will no longer be accepted by SPEC for publication.

Q.) Can SPEC SFS 2.0 results be compared to SFS 1.1 results?

A.) No. Although the benchmarks are similar, they cannot be compared, since SFS 2.0 uses a different workload.

Q.) What improvements have been made to SPEC SFS 2.0?

A.) In addition to general code improvements, SPEC SFS 2.0 includes five major enhancements: 1. It measures NFS protocol version 3 results in addition to those from NFS protocol version 2. 2. It adds support for TCP; either TCP or UDP can be used as the network transport. 3. SPEC SFS 2.0's NFS operation mix more closely matches today's real-world NFS loads. 4. The benchmark distribution CD contains pre-compiled and tested binaries. 5. It has an improved interface to accommodate both accomplished and novice users.

Q.) How was the SPEC SFS 2.0 workload determined?

A.) The SPEC SFS 2.0 workload is based primarily on a survey of more than 1,000 servers in different application environments. The survey found that 60 percent of these users have similar mixes of NFS operations.

Q.) What is the metric for SPEC SFS 2.0?

A.) SPEC SFS 2.0 has two performance measurement metrics: SPECsfs97.v2 for NFS protocol version 2 and SPECsfs97.v3 for NFS protocol version 3. Both metrics include a throughput measure (in operations per second) and an overall response time measure (the average response time per operation).

Q.) Are the metrics for SPEC SFS 2.0 different than the metric for SFS 1.1?

A). Yes. SFS 2.0 removes the SFS 1.1 metric for response time at peak measured throughput and replaces it with the overall response time and peak throughput. The larger the peak throughput the better. The lower the overall response time the better. The overall response time is an indicator of how quickly the system under test responds to NFS operations over the entire range of the tested load. In real-world situations, servers are not run continuously at peak throughput, so peak response time provides only minimal information. The overall response time is a measure of how the system will respond under an average load. Mathematically, the value is derived by calculating the area under the curve divided by the peak throughput. Below the first data point is assumed to have a constant response time equal to that of the first data point.

Q.) How widespread is NFS version 3?

A.) NFS version 3 has been shipping on systems for more than three years and is available for most systems that support NFS version 2.

Q.) What is the correlation between the TPC (Transaction Processing Council) benchmarks and SPEC SFS 2.0?

A.) There is no correlation; the benchmarks measure totally different aspects of system performance.

Q.) Is SPEC SFS 2.0 a CPU- or I/O-intensive benchmark?

A.) SPEC SFS 2.0 is a system-level benchmark that heavily exercises CPU, mass storage and network components. The greatest emphasis is on I/O, especially as it relates to operating and file system software. To obtain the best performance for a system running SFS 2.0, the vendor will typically add additional hardware -- such as memory, disk controllers, disks, network controllers and buffer cache -- to help alleviate I/O bottlenecks and to ensure that server CPUs are used fully.

Q.) For what computing environment is SPEC SFS 2.0 designed?

A.) The benchmark was developed for load-generating clients running in the UNIX environment. But since the load-generating clients execute the benchmark code, SPEC SFS 2.0 can be used to test the performance of any NFS server, regardless of the underlying environment. Porting is required, however, for non-UNIX environments.

Q.) Can users measure NFS performance for workloads other than the one provided within SPEC SFS 2.0?

A.) Yes, users can measure their own workloads by making changes to the SPECsfs97 benchmark mix parameters to reflect the new measurements. The SPEC SFS 2.0 User's Guide details how this can be done. Workloads created by users cannot, however, be compared with SFS 2.0 results, nor can they be published in any form, as specified within the SFS 2.0 license.

Q.) To what extent is the server's measured performance within SPEC SFS 2.0 affected by the client's performance?

A.) SPEC has written SFS 2.0 to minimize the effect of client performance on SPECsfs97 results.

Q.) Why have only three companies reported SPECsfs97 results in conjunction with this announcement?

A.) SPEC SFS 2.0 is a system-level benchmark that requires scheduling substantial resources for testing. SPEC expects other member companies to report results in the near future.

Q.) How does SPEC validate numbers that it publishes?

A.) Results published on the SPEC Web site and in the SPEC newsletter have been reviewed by SPEC members for compliance with the SFS 2.0 run and disclosure rules, but there is no monitoring beyond that compliance check. The vendors that performed the tests and submitted the performance numbers have sole responsibility for the results. SPEC is not responsible for any measurement or publication errors.

Q.) Are the reported SFS 2.0 configurations typical of systems sold by vendors?

A.) Yes and no. They are similar to large server configurations, but the workload is heavier than that found on smaller server configurations. SPEC has learned from experience that today's heavy workload is tomorrow's light workload. For some vendors, the configurations are typical of what they see in real customer environments, particularly those incorporating high-end servers. For other vendors, SFS 2.0 configurations might not be typical.

Q.) Do the SFS 2.0 run and disclosure rules allow results for a clustered server?

A.) Yes, cluster configurations are allowed as long as they conform strictly to the even distribution of all resources as defined by the SFS 2.0 run and disclosure rules.

Q.) Why do so few published results approach SPEC's response-time threshold cutoff of 40 milliseconds?

A.) It is important to understand first that SPECsfs97 run rules do not require that the throughput curve be carried out to 40 ms; they only state that the results cannot be reported for a response time higher than 40 ms. There are several reasons why results do not approach the threshold cutoff. Optimally configured servers often will achieve their maximum throughput at response times lower than the cutoff. Additionally, some vendors emphasize maximum throughput while others concentrate on fast response time. It does not indicate a problem with the results if the curve is not carried out to 40 ms, and those reviewing results should not try to predict what the throughput curve might be past the reported point.

Q.) Why was the response-time threshold reduced from 50 ms for SFS 1.1 to 40 ms for SFS 2.0?

A.) The lower response-time threshold reflects advances in server technologies since the release of SFS 1.1 in January 1995.

Q.) What resources are needed to run the SPEC SFS 2.0 benchmark?

A.) In addition to a server, a test bed includes several clients and an appropriate number of networks. The server must have enough memory, disks and network hardware to saturate the CPU. The test bed requires at least one network and each network must have sufficient client capacity to saturate the network(s). A minimum of 64 MB of memory is required for each client, although in most cases 128 MB is needed. Requirements are detailed in the SFS 2.0 User's Guide. To facilitate accuracy of reported vendor results, SFS 2.0 includes an entire NFS implementation. Examples of typical load-generating configurations can be found on the SPEC Web site.

Q.) What is the estimated time needed to set up and run SPEC SFS 2.0?

A.) Hardware setup and software installation time depend on the size of the server and the complexity of the test beds. Many servers require large and complex test beds. The SFS 2.0 software installs relatively quickly. A SPECsfs97 submission from a vendor includes at least 10 data points, with each data point taking about 20 to 30 minutes to complete.

Q.) What shared resources does SPEC SFS 2.0 use that might limit performance?

A.) Shared resources that might limit performance include disk controllers, disks, network controllers, network concentrators, network switches and clients.

Q.) SPEC's CPU95 benchmark defines compiler optimization flags that can be used in testing. Does SPEC SFS 2.0 set tuning parameters?

A.) When submitting results for SPEC review, vendors are required to supply a description of all server tuning parameters within the disclosure section of the reporting page.

Q.) Can a RAM disk be used within a SPEC SFS 2.0 configuration?

A.) SPEC enforces strict storage rules for stability. Generally, RAM disks do not meet these rules, since they often cannot survive cascading failure-recovery requirements unless an uninterruptible power supply (UPS) with long survival lines is used.

Q.) How will the choice of networks affect SFS 2.0 results?

A.) Different link types and even different implementations of the same link type might affect the measured performance -- for better or worse -- of a particular server. Consequently, the results measured by clients in these situations might vary as well.

Q.) Is SPEC SFS 2.0 scalable with respect to CPU, cache, memory, disks, controllers and faster transport media?

A.) Yes, like SFS 1.1, the new benchmark is scalable as users migrate to faster technologies.

Q.) What is the price of a SPEC SFS 2.0 license and when will it be available?

A.) SPEC SFS 2.0 is available now on CD-ROM for $900. Contact the SPEC office: SPEC, 10754 Ambassador Dr., Ste. 201, Manassas, VA 20109; tel: 703-331-0180; fax: 703-331-0181; email: info@spec.org.

Q.) How much is an upgrade from SFS 1.1 to SFS 2.0?

A.) The upgrade is free for those who have purchased SFS 1.1 licenses within the last three months and $300 for other SFS 1.1 licensees. Upgrades are available through the SPEC office.

Q.) Can users get help in running SPEC SFS 2.0?

A.) The majority of questions should be answered in the SPEC SFS 2.0 User's Guide. There is also useful information on the SPEC Web site.