SPEC CPU®2017 Overview / What's New?

A surveyor's bench mark (two words) defines a known reference point by which other locations may be measured.
A computer benchmark performs a known set of operations by which computer performance can be measured.

Yes, if you can; but it may be be difficult.

Ideally, the best comparison test for systems would be your own application with your own workload. Unfortunately, it is often impractical to get a wide set of comparable system measurements using your own application with your own workload. For example, it may be difficult to extract the application sections that you want to benchmark, or too difficult to remove confidential information from data sets.

It takes time and effort to create a good benchmark, and it is easy to fall into common mistakes.

Maybe. A standardized benchmarks may provide a reference point, if you use it carefully.

You may find that a standardized benchmark has already been run on systems that you are interested in. Ideally, that benchmark will provide all the characteristics of Table 1 while avoiding common benchmark mistakes.

Before you consider the results of a standardized benchmark, you should consider whether it measures things that are important to your own application characteristics and computing needs. For example, a benchmark that emphasizes CPU performance will have limited usefulness if your primary concern is network throughput.

A standardized benchmark can serve as a useful reference point, but SPEC does not claim that any standardized benchmark can replace benchmarking your own actual application when you are selecting vendors or products.

SPEC CPU 2017 focuses on compute intensive performance, which means these benchmarks emphasize the performance of:

• Processor - The CPU chip(s).
• Memory - The memory hierarchy, including caches and main memory.
• Compilers - C, C++, and Fortran compilers, including optimizers.

SPEC CPU 2017 intentionally depends on all three of the above - not just the processor.

SPEC CPU 2017 is not intended to stress other computer components such as networking, graphics, Java libraries, or the I/O system. Note that there are other SPEC benchmarks that focus on those areas.

Briefly, you will need a computer running Linux, Mac OS X, Unix, or Microsoft Windows with:

• 1 to 2 GB of main memory per copy, if you want to run SPECrate suites; or
  16 GB to run SPECspeed suites.
• 250 GB disk space is recommended; a minimal installation needs 10 GB.
• C, C++, and Fortran compilers (or a set of pre-compiled binaries from another CPU 2017 user).
• A variety of chips and operating systems are supported.

The above is only an abbreviated summary. See detail in the System Requirements document.

New with CPU 2017, the integer suites include a Fortran benchmark.
Reportable runs for all suites require compilation of all three languages (C, C++, Fortran).

A one-page summary is in SPEC CPU 2017 Quick Start. Here is a summary of the summary:

• Meet the System Requirements.
• Install SPEC CPU 2017 [Unix] [ Windows]
• Find a config file -- a file that defines how to build, run, and report on the SPEC CPU benchmarks in a particular environment -- and customize it.
• Determine which suite you wish to run.
• Use the runcpu command.
• If you wish to generate results suitable for quoting in public, you will need to carefully study and adhere to the Run Rules.

A suite is a set of benchmarks that are run as a group to produce one of the overall metrics.

SPEC CPU 2017 includes four suites that focus on different types of compute intensive performance:

Most of the benchmarks in the table above are part of a pair:

 5nn.benchmark_r for the SPECrate version
 6nn.benchmark_s for the SPECspeed version

Benchmarks within a pair are similar to each other.
Briefly, differences include: workload sizes; compile flags; and run rules. See: [memory]   [OpenMP]   [rules]

More detail:

• Memory Usage, -m32 and -m64
  Although the names are similar, the workloads often differ.
  For SPECrate, you may be able to choose between 32-bit or 64-bit compilation (check your compiler documentation for flags such as -m32 or -m64).
  For SPECspeed, you will usually need 64 bits (-m64).

• OpenMP directives: never for SPECrate
  OpenMP directives are always suppressed (not visible to the compiler) when building the SPECrate benchmarks.

• OpenMP directives: optional for SPECspeed
  OpenMP directives are optionally available for all SPECspeed 2017 Floating Point benchmarks, and for one SPECspeed 2017 Integer benchmark, 657.xz_s. [OpenMP]

• Prohibited flags
  For SPECrate, compiler parallelization is forbidden -- both OpenMP and compiler autopar.

• Other differences
  Some pairs enable different source via compile flags (which are automatically provided by the SPEC CPU toolset). For example, 627.cam4_s includes the flag -DUSE_COSP to enable a different method of cloud modeling than is used for 527.cam4_r; and the Chess program deepsjeng is compiled with flags to tell it which size it should use for its primary data structures (-DSMALL_MEMORY or -DBIG_MEMORY).

There are many ways to measure computer performance. Among the most common are:

• Time - For example, seconds to complete a workload.
• Throughput - Work completed per unit of time, for example, jobs per hour.

SPECspeed is a time-based metric; SPECrate is a throughput metric.

time on a reference machine / time on the SUT

number of copies * (time on a reference machine / time on the SUT)

For the energy metrics, the calculations are done the same way, using energy instead of time in the above formulas.

The reference times and reference energy are based on the observations posted with www.spec.org/cpu2017/results/

1, 2, 3, and 4

The measured values, with all available digits, can be found in the CSV versions:

1, 2, 3, and 4.

The reference times used in the calculations are rounded to the next higher integral number of seconds.

SPEC CPU benchmarks are distributed as source code, and must be compiled, which leads to the question:
How should they be compiled? There are many possibilities, ranging from

--debug --no-optimize

at a low end through highly customized optimization and even source code re-writing at a high end. Any point chosen from that range might seem arbitrary to those whose interests lie at a different point. Nevertheless, choices must be made.

For CPU 2017, SPEC has chosen to allow two points in the range. The first may be of more interest to those who prefer a relatively simple build process; the second may be of more interest to those who are willing to invest more effort in order to achieve better performance.

• The base metrics (such as SPECspeed2017_int_base) require that all modules of a given language in a suite must be compiled using the same flags, in the same order. All reported results must include the base metric.

• The optional peak metrics (such as SPECspeed2017_int_peak) allow greater flexibility. Different compiler options may be used for each benchmark, and feedback-directed optimization is allowed.

Options allowed under the base rules are a subset of those allowed under the peak rules. A legal base result is also legal under the peak rules but a legal peak result is NOT necessarily legal under the base rules.

For more information, see the SPEC CPU 2017 Run and Reporting Rules.

1. Benchmarks and Metrics

2. Source Code: C99, Fortran-2003, C++2003

Total source code has increased, as shown in the graph [larger version], because most benchmarks are derived from real applications (including various open source projects). During benchmark development, SPEC spends substantial effort working to improve portability, using language standards to assist in the process. For CPU 2017, the standards referenced were C99, Fortran-2003, and C++2003. Caution: The benchmarks do not comply perfectly with ISO/ANSI lanaguage standards, because their source code is derived from real applications. The rules allow optimizers to assume standards *only* where that does not prevent validation.

3. Rules

• Rule 1.2.1: Reportable runs can use the median, or the slower of two runs
• Rule 2.3.1: The standards referenced are C99, Fortran-2003, C++2003.
• Rule 2.3.9: Data models: expand to clarify treatment of 64-bit file data
• Rule 3.4 allows the peak run-time environment to differ by benchmark.
• Rule 3.6 and 3.7: for both copies and threads, base must be the same, peak can differ.
• Rule 4.1.2: The sysinfo utility is required for reportable runs
• More detail

SPEC CPU 2017 includes the ability to optionally measure and report power, including maximum power (W), average power (W), and total energy used (kJ).

Power measurement and reporting for CPU 2017 is optional: Testers are not required to measure power.

If you wish to measure power, you will need:

• A power analyzer.
• A temperature sensor.
• A Linux or Windows "Controller System", where you will run SPEC's Power and Temperature reporting tool, SPEC PTDaemon™

Metrics: Details of the calculation of the energy metrics can be found here.
New with SPEC CPU 2017 v1.1, the energy metrics are no longer labelled "experimental". They are fully supported and comparable.

For more information, see the config file documentation on Power Measurement.

There is now a 1-page Quick Start.

There is a collection of working examples in the top-level config directory, with working flags files.

Most documents have been updated for CPU 2017. A few highlights of note:

Make Variables has a handy table of which variables affect which build steps.

SPEC CPU 2017 Config Files:
Reorganized for clarity.
Examples have been tested, and many of them are on your installed copy of SPEC CPU 2017.

1. SPECrate smarter setup As systems become larger, and as chips, cores, and hardware threads become smaller, SPECrate runs are asked to handle many more files. For SPEC CPU 2006, runs with hundreds of copies would take hours to set up, requiring around 750 MB (million bytes, 10^6) of disk space for each additional copy after the first copy (setup only, prior to actually running benchmarks; base only; set up both integer and floating point). For SPEC CPU 2017, setup typically takes a few minutes, and each additional copy after the first requires about 200 KB (kilobytes, 10^3). An example system doing SPECrate setup to a single spinning hard drive is shown in the graphs on the right [larger version]. The additional space and time needed per copy for SPEC CPU 2017 is so small as to be barely visible (bottom two lines, in blue). Of course, IO rates are subject to substantial variability according to operating system, file system, and storage type; and may hit "knees in the curve", as in the red line in the bottom graph, for various reasons. This particular test happened to hit a file system cache limit. q. Why is SPECrate 2017 setup faster and smaller? a. All input files are hard linked to the first copy. The concept of hard links (Wikipedia) is commonly supported on modern file systems. Obviously, this optimization only helps benchmark setup, not benchmark output; which is why System Requirements recommends that you allow substantial disk space per copy. q. What if my file system lacks hard links? a. If a SPECrate run is attempted on a file system that does not support hard links, the tools will fall back to the old method, making multiple copies. To check whether this happens, search the log for a message of the form: NOTICE: Linking file from srcdir failed ...[additional text]... Switching to copy mode. If you set verbose or --verbose to 10 or higher, the NOTICE will appear on your screen.

2. sysinfo is required

For SPEC CPU 2017, results that are used in public must use sysinfo, which creates a contemporaneous record of the system under test (automatically, and therefore not subject to human transcription error). [details]

3. Compiler versions are required

SPEC CPU performance depends on compilers. For CPU 2017, all config files must include:

CC_VERSION_OPTION   = (switch that causes version to be printed)
CXX VERSION_OPTION  = "       "    "      "       "  "  "
FC_VERSION_OPTION   = "       "    "      "       "  "  "

Builds will fail until you add the above to your config file.
Your installed copy of SPEC CPU 2017 includes Example config files that demonstrate what to include for a variety of compilers. [detail]

4. Image Validation

Several benchmarks in CPU 2017 generate images. In order to determine whether acceptable images are generated, SPEC CPU 2017 validates the output using the Structured SIMilarity method, producing an Average Similarity (AVG SSIM). The image validator is described in SPEC CPU 2017 Utilities.

Are both of these 'correct'? Sometimes the answer is obvious. AVG SSIM=0.661

Sometimes, it's a little harder to decide AVG SSIM=0.998

5. Usability

default:

default=default=default:

6. Removed items

• The parallel_setup features have been removed because setup is so much faster now. [detail].
• The --rate and --speed switches have been removed, because speed and rate mode are now tied to the benchmark (5xx or 6xx). [detail].
• runspec is gone The former runspec utility is renamed runcpu in SPEC CPU 2017.   [Why?]

Yes, SPEC CPU 2017 results can be published independently, and Yes, the rules still apply.

• The SPEC CPU license requires that published results must follow the SPEC CPU 2017 Run and Reporting Rules.
• Any public use of SPEC benchmark results is subject to the SPEC Fair Use Rule, which requires (among other things) that your statement must be clear; must be correct; must reference public data; must identify data sources; must clearly distinguish between SPEC metrics and other types of data.
• Your results must comply with any rule updates that may be posted from time to time at
  www.spec.org/cpu2017/Docs/runrules.html.
• Any SPEC member may request the rawfile from your result, per Rule 5.5.

Although you are allowed to publish indpendently, SPEC encourages results to be submitted for publication on SPEC's web site, because it ensures a peer review process and uniform presentation of all results.

The Fair Use rule recognizes that Academic and Research usage of the benchmarks may be less formal; the key requirement is that non-compliant numbers must be clearly distinguished from rule-compliant results.

SPEC CPU performance results may be estimated, but SPEC CPU energy metrics are not allowed to be estimated. Estimates must be clearly marked. For more information, see the SPEC CPU 2017 rule on estimates and the CPU 2017 section of the general SPEC Fair Use rule.