SPEC OMP2001 Readme First

Last updated: 14 Mar 2001 17:00pm wesley@sgi.com
(To check for possible updates to this document, please see http://www.spec.org/hpg/omp2001 )

Contents

Documentation Overview

Location:

Prior to installation, look on your CD in docs/ or docs.nt\ -- for example, if your CD is mounted on /cdrom (Unix) or drive E: (NT), you would find most of the documents in:

/cdrom/docs/ (Unix) or 
        E:\docs.nt\  (NT)
After installation, look in:
$SPEC/docs/     (Unix) or
        %SPEC%\docs.nt\ (NT)
Suggested reading order:

If you are just getting started with SPEC OMP2001, here is a suggested reading order for the various documents supplied by SPEC:

  1. readme1st.html (the document you are reading now)
  2. system_requirements.html
  3. install_guide_unix.html or install_guide_nt.html
  4. runspec.html
  5. other documents as needed
List of documents:

The following documents are located in docs/ or docs.nt\ as described above under "Location:"

config.html To run SPEC OMP2001, you need a config file. This document tells you how to write one.
errata.txt Debugging and errata information.
example-advanced.cfg A complex sample config file with commentary.
example-medium.cfg A complete, but not very complex, sample config file with commentary.
example-simple.cfg A simple configuration file with commentary. A first time user could take this as a template for a first run of SPEC OMP2001.
install_guide_unix.html How to install SPEC OMP2001 on UNIX systems. Includes an example installation and an example of running the first benchmark.
install_guide_nt.txt How to install SPEC OMP2001 on Windows/NT systems. Includes an example installation and an example of running the first benchmark.
legal.txt Copyright notice and other legal information.
makevars.txt Advanced users of the suite who want to understand exactly how the benchmarks are built can use this file to help decipher the process.
readme1st.html The document you are reading now, contains a documentation overview and question and answers regarding the purpose and intent of SPEC OMP2001.
runrules.html The SPEC OMP2001 Run and reporting rules. These must be followed for generating publicly disclosed results.
runspec.html Information on the "runspec" command, which is the primary user interface for running SPEC OMP2001 benchmarks.
system_requirements.html A list of the hardware and software needed in order to run the SPEC OMP2001 suite.
techsupport.txt Information on SPEC technical support.
tools_build.txt How to build (or re-build) the tools such as runspec
utility.html How to use various utilities, such as specinvoke and specdiff

In addition, each individual benchmark in the suite has its own documents, found in the benchmark "docs" subdirectory. For example, the description of the benchmark 330.art_m may be found in:

  $SPEC/benchspec/OMPM2001/330.art_m/docs/330.art_m.txt (Unix) or
    %SPEC%\benchspec\OMPM2001\330.art_m\docs\330.art_m.txt (NT)

Only on the CD, you will find:

 original.src/README   Information about freely-available sources
                          that have been incorporated in SPEC OMP2001

SPEC OMP2001 Overview

Background

By providing this background, SPEC hopes to help the user set their expectations and usage appropriately to get the most efficient and beneficial use out of this benchmark product.

Overall, SPEC designed SPEC OMP2001 to provide a comparative measure of shared memory processor (SMP) performance across the widest practical range of hardware. This resulted in source code benchmarks developed from real user applications. These benchmarks are dependent on the processor, memory, compiler, and OpenMP implementation on the tested system.

Q1. What is SPEC?

SPEC is an acronym for the Standard Performance Evaluation Corporation. SPEC is a non-profit organization composed of computer vendors, systems integrators, universities, research organizations, publishers and consultants whose goal is to establish, maintain and endorse a standardized set of relevant benchmarks for computer systems. Although no one set of tests can fully characterize overall system performance, SPEC believes that the user community will benefit from an objective series of tests which can serve as a common reference point.

Q2. What is a benchmark?

The definition from Webster's II Dictionary states that a benchmark is "A standard of measurement or evaluation."

A computer benchmark is typically a computer program that performs a strictly defined set of operations (a workload) and returns some form of result (a metric) describing how the tested computer performed. Computer benchmark metrics usually measure speed (how fast was the workload completed) or throughput (how many workloads per unit time were measured). Running the same computer benchmark on multiple computers allows a comparison to be made.

Q3. Why use a benchmark?

Ideally, the best comparison test for systems would be your own application with your own workload. Unfortunately, it is often very difficult to get a wide base of reliable, repeatable and comparable measurements for comparisons of different systems on your own application with your own workload. This might be due to time, money, confidentiality, or other constraints.

Q4. What options are viable in this case?

At this point, you can consider using standardized benchmarks as a reference point. Ideally, a standardized benchmark will be portable and maybe already run on the platforms that you are interested in. However, before you consider the results you need to be sure that you understand the correlation between your application/computing needs and what the benchmark is measuring. Are the workloads similar and do they have the same characteristics? Based on your answers to these questions, you can begin to see how the benchmark may approximate your reality.

Note: It is not intended that the SPEC benchmark suites be used as a replacement for the benchmarking of actual customer applications to determine vendor or product selection.

Q5. What does SPEC OMP2001 measure?

SPEC OMP2001 focuses on SMP performance, which means these benchmarks emphasize the performance of:

It is important to remember the contribution of the latter two components; performance is more than just the processor.

SPEC OMP2001 is made up of two subcomponents that focus on size of the SMP parallel systems::

Note that SPEC OMP2001 does not stress other computer components such as I/O (disk drives), networking, operating system or graphics. It might be possible to configure a system in such a way that one or more of these components impact the performance of OMPM2001 and OMPL2001, but that is not the intent of the suites.

Q6. Why use SPEC OMP2001?

As mentioned above, SPEC OMP2001 provides a comparative measure of medium and large SMP performance. If this matches with the type of workloads you are interested in, SPEC OMP2001 provides a good reference point.

Other advantages to using SPEC OMP2001:

Q7. What are the limitations of SPEC OMP2001?

As described above under "Why use a benchmark?", the ideal benchmark for vendor or product selection would be your own workload on your own application. Please bear in mind that no standardized benchmark can provide a perfect model of the realities of your particular system and user community.

Q8. What is included in the SPEC OMP2001 package?

SPEC provides the following on the SPEC OMP2001 Media:

Q9. What does the user of the SPEC OMP2001 suite have to provide?

Briefly, you need a Unix or NT system with 2 or 8 GB of memory for OMPM2001 and OMPL2001 respectively, 4GB of disk, and a set of compilers. Please see the details in the file system_requirements.html

Q10. What are the basic steps in running the benchmarks?

Installation and use are covered in detail in the SPEC OMP2001 User Documentation. The basic steps are as follows:

Q11. What source code is provided? What exactly makes up these suites?

OMPM2001 (Medium) and OMPL2001 (Large) are based on compute-intensive applications provided as source code.

OMPM2001 contains 9 applications written in Fortran and 2 in C that are used as benchmarks:

Name Remarks
310.wupwise_m Quantum chromodynamics
312.swim_m Shallow water modeling
314.mgrid_m Multi-grid solver in 3D potential field
316.applu_m Parabolic/elliptic partial differential equations
318.galgel_m Fluid dynamics: analysis of oscillatory instability
320.equake_m Finite element simulation; earthquake modeling
324.apsi_m Solves problems regarding temperature, wind, velocity and distribution of pollutants
326.gafort_m Genetic algorithm
328.fma3d_m Finite element crash simulation
330.art_m Neural network simulation; adaptive resonance theory
332.ammp_m Computational Chemistry

OMPL2001 contains 9 of the same 11 applications with larger data sets that are used as benchmarks: 311.wupwise_l, 313.swim_l, 315.mgrid_l, 317.applu_l, 321.equake_l, 325.apsi_l, 327.gafort_l, 329.fma3d_l, 331.art_l.

More detailed descriptions on the benchmarks (with reference to papers, web sites, etc.) can be found in the individual benchmark directories in the SPEC benchmark tree.

The numbers used as part of the benchmarks names provide an identifier to help distinguish programs from one another. For example, some programs were updated from SPEC CPU2000 and need to be distinguished from their previous version.

Q12. Some of the benchmark names sound familiar; are these comparable to other programs?

Many of the SPEC benchmarks have been derived from publicly available application programs and all have been developed to be portable to as many current and future hardware platforms as practical. Hardware dependencies have been minimized to avoid unfairly favoring one hardware platform over another. For this reason, the application programs in this distribution should not be used to assess the probable performance of commercially available, tuned versions of the same application. The individual benchmarks in this suite may be similar, but NOT identical to benchmarks or programs with the same name which are available from sources other than SPEC; therefore, it is not valid to compare SPEC OMP2001 benchmark results with anything other than other SPEC OMP2001 benchmark results. (Note: This also means that it is not valid to compare SPEC OMP2001 results to older SPEC CPU benchmarks; these benchmarks have been changed and should be considered different and not comparable.)

Q13. What metrics can be measured?

The OMPM2001 and OMPL2001 suites can be used to measure and calculate the following metrics:

OMPM2001 (for medium Compute intensive SMP performance comparisons):
- SPECompMpeak2001: The geometric mean of eleven normalized ratios (one for each medium benchmark) when compiled with aggressive optimization for each benchmark.
- SPECompMbase2001: The geometric mean of eleven normalized ratios when compiled with conservative optimization for each benchmark.
OMPL2001 (for large Compute intensive SMP performance comparisons):
- SPECompLpeak2001: The geometric mean of eleven normalized ratios (one for each large benchmark) when compiled with aggressive optimization for each benchmark.
- SPECompLbase2001: The geometric mean of eleven normalized ratios when compiled with conservative optimization for each benchmark.

The ratio for each of the benchmarks is calculated using a SPEC- determined reference time and the run time of the benchmark.

A higher score means "better performance" on the given workload.

Q14. What is the difference between a "base" metric and a "peak" metric?

In order to provide comparisons across different computer hardware, SPEC provides the benchmarks as source code. Thus, in order to run the benchmarks, they must be compiled. There is agreement that the benchmarks should be compiled the way users compile programs. But how do users compile programs?

Some people might experiment with many different compilers and compiler flags to achieve the best performance. Other people might just compile with the basic options suggested by the compiler vendor. SPEC recognizes that it cannot exactly match how everyone uses compilers, but two reference points are possible:

Note that the base metric rules are a subset of the peak metric rules. For example, a legal base metric is also legal under the peak rules but a legal peak metric is NOT legal under the base rules.

A full description of the distinctions and required guidelines can be found in the SPEC OMP2001 Run and Reporting Rules available with SPEC OMP2001.

Q15. Which SPEC OMP2001 metric should be used to compare performance?

It depends on your needs. SPEC provides the benchmarks and results as tools for you to use. You need to determine how you use a computer or what your performance requirements are and then choose the appropriate SPEC benchmark or metrics.

Q16. How do I contact SPEC for more information or for technical support?

SPEC can be contacted in several ways. For general information, including other means of contacting SPEC, please see SPEC's World Wide Web Site at http://www.spec.org/

General questions can be emailed to: info@spec.org

OMP2001 Technical Support Questions can be sent to: OMP2001support@spec.org

Q17. Now that I've read this document, what should I do next?

You should verify that your system meets the requirements as described in

system_requirements.html

and then you can install the suite, following the instructions in

install_guide_unix.html or install_guide_nt.html