SPEC OMP2012
Run and Reporting Rules
SPEC High Performance Group

3.2 Controlling Benchmark Jobs

3.2.1 Number of runs in a reportable result

A reportable run consists of three runs of the suite. The reportable result will be the median of these three runs.

3.2.2 Number of threads in base

For SPECompG_base2012 measurements, the tester must select a single value to use as the number of threads to be applied to all benchmarks in the suite.

3.2.3 Number of threads in peak

For SPECompG_peak2012, the tester is free to choose the number of threads for each individual benchmark independently of the other benchmarks, and this number may be less than, equal to, or greater than the number of threads specified for base.

3.2.4 The submit directive

The config file directive submit is the preferred means to assign work to processors. The tester may, if desired:

1. place benchmarks on desired processors;
2. place the benchmark memory on a desired memory unit;
3. do arithmetic (e.g. via shell commands) to derive a valid processor number for each thread;
4. cause the tools to write each copy's benchmark invocation lines to a file, which is then sent to its processor;
5. reference a testbed description provided by the tester.

The submit directive can be used to change the run time environment (see section 3.4). In addition, if a testbed description is referenced by a submit directive, the same description must be used by all benchmarks in a base run. This means that in base, the submit directive may only differ between benchmarks in the suite for portability reasons.

In peak, different benchmarks may use different submit directives.

3.3 Power Measurement

The system configuration used for the measurement of power must be in accordance with Section 3.1.3 and the power measurement itself must be in accordance with the SPEC Power and Performance Benchmark Methodology, version 2.1 (08/17/2011). The SPEC OMP2012 benchmark tool set provides the ability to automatically gather measurement data from supported power analyzers and temperature sensors and integrate that data into the benchmark result. SPEC requires that the analyzers and sensors used in a submission be supported by the measurement framework, and be compliant with the specifications in Section 3.1.3. The tools provided by SPEC OMP2012 for power measurement (namely PTDaemon), or a more recent version provided by SPECpower must be used to run and produce measured SPEC OMP2012 results. SPEC OMP2012 includes PTDaemon version 1.4.2. For the latest version of the PTDaemon and for the list of accepted measurement devices, see this page.

3.4 Run-time Environment

Run-time environment settings are treated similarly to compilation options in SPEC OMP2012. The rules are as follows, from highest precedence to lowest:

1. Environment settings may be written into the submit line of the config file, i.e.

   submit = export MP_SPIN=wait; ....
   

   Settings are hard to discern if too many details are packed into the submit line. One advantage, however, is that changing the settings does not cause anything to rebuild.
2. Environment settings may be made in the config file using entries of the form

   env_vars = 1
   ENV_MP_SPIN = wait
   

   The settings are quite clear from reading the text of the config file. A disadvantage is that the settings also apply to the build phase, and changing a setting will cause the affected benchmarks to rebuild.
3. Environment settings may be made outside the invocation of runspec:

   export MP_SPIN=...
   runspec ...
   

   The settings are invisible to the automatic report generation and must be carefully documented.
4. For base runs, environment settings should be consistent for all benchmarks in the suite. If this is not possible, the same rules apply as with portability flags in how the committee will accept them.
5. Environment settings can be varied between benchmarks in peak runs. Note that you can't do this with type 3 settings made prior to the runspec invocation.
   
6. The semantics of the settings must be documented.

These SPEC OMP2012 run-time environment rules are consistent with MPI2007 but deviate from CPU2000 and CPU2006 because environment settings affecting the OpenMP and MPI libraries play an essential role in application performance.

3.5 Continuous Run Requirement

All benchmark executions, including the validation steps, contributing to a particular submittable report must occur continuously, that is, in one execution of runspec. For a reportable run, the runspec tool will run all three workloads (test, train, and ref), and will ensure that the correct answer is obtained for all three. (Note: the execution and validation of test and train is not part of the timing of the benchmark - it is only an additional test for correct operation of the binary.)

3.6 Base, Peak, and Basepeak

If a submittable report will contain both base and peak measurements, a single runspec invocation must be used for the runs. When both base and peak are run, the tools run the base executables first, followed by the peak executables.

It is permitted to publish base results as peak. This can be accomplished in various ways, all of which are allowed:

1. Set basepeak=yes in the config file for individual benchmarks.

2. Set basepeak=yes in the config file for an entire suite.

3. Select the --basepeak option when using rawformat.

Note: It is permitted but not required to compile in the same runspec invocation as the execution. See Section 2.1.5 regarding cross compilation.

3.7 Run-Time Dynamic Optimization

3.7.1 Definitions and Background

As used in these run rules, the term "run-time dynamic optimization" (RDO) refers broadly to any method by which a system adapts to improve performance of an executing program based upon observation of its behavior as it runs. This is an intentionally broad definition, intended to include techniques such as:

• rearrangement of code to improve instruction cache performance
• replacement of emulated instructions by native code
• value prediction
• branch predictor training
• reallocation of on-chip functional units among hardware threads
• TLB training
• adjustment of the supply of big pages

RDO may be under control of hardware, software, or both.

Understood this broadly, RDO is already commonly in use, and usage can be expected to increase. SPEC believes that RDO is useful, and does not wish to prevent its development. Furthermore, SPEC views at least some RDO techniques as appropriate for base, on the grounds that some techniques may require no special settings or user intervention; the system simply learns about the workload and adapts.

However, benchmarking a system that includes RDO presents a challenge. A central idea of SPEC benchmarking is to create tests that are repeatable: if you run a benchmark suite multiple times, it is expected that results will be similar, although there will be a small degree of run-to-run variation. But an adaptive system may recognize the program that it is asked to run, and "carry over" lessons learned in the previous execution; therefore, it might complete a benchmark more quickly each time it is run. Furthermore, unlike in real life, the programs in the benchmark suite are presented with the same inputs each time they are run: value prediction is too easy if the inputs never change. In the extreme case, an adaptive system could be imagined that notices which program is about to run, notices what the inputs are, and which reduces the entire execution to a print statement. In the interest of benchmarking that is both repeatable and representative of real-life usage, it is therefore necessary to place limits on RDO carry-over.

3.7.2 RDO Is Allowed, Subject to Certain Conditions

Run time dynamic optimization is allowed, subject to the usual provisions that the techniques must be generally available, documented, and supported. It is also subject to the conditions listed in the rules immediately following.

3.7.3 RDO Disclosure and Resources

Section 4.2 applies to run-time dynamic optimization: any settings which the tester has set to non-default values must be disclosed. Resources consumed by RDO must be included in the description of the hardware configuration as used by the benchmark suite.

3.7.4 RDO Settings Cannot Be Changed At Run-time

Run time dynamic optimization is subject to Section 3.4: settings cannot be changed at run-time. But Note 2 of rule 3.4 also applies to RDO: for example, in peak it would be acceptable to compile a subset of the benchmarks with a flag that suggests to the run-time dynamic optimizer that code rearrangement should be attempted. Of course, Section 2.2.1 also would apply: such a flag could not tell RDO which routines to rearrange.

3.7.5 RDO and safety in base

If run-time dynamic optimization is effectively enabled for base (after taking into account the system state at run-time and any compilation flags that interact with the run-time state), then RDO must comply with 2.3.1, the safety rule. It is understood that the safety rule has sometimes required judgment, including deliberation by SPEC in order to determine its applicability. The following is intended as guidance for the tester and for SPEC:

• If an RDO system optimizes a SPEC benchmark in a way which allows it to successfully process the SPEC-supplied inputs, that is not enough to demonstrate safety. If it can be shown that a different, but valid, input causes the program running under RDO to fail (either by giving a wrong answer or by exiting), where such failure does not occur without RDO, and if it is not a fault of the original source code, then this is taken as evidence that the RDO method is not safe.
• If an RDO system requires that programs use a subset of the relevant ANSI/ISO language standard, or requires that they use non-standard features, then this is taken as evidence that it is not safe.
• But an RDO system is allowed to assume that the programs adhere to the relevant ANSI/ISO language standard.

3.7.6 RDO carry-over by program is not allowed

As described in section 3.7.1, SPEC has an interest in preventing carry-over of information from run to run. Specifically, no information may be carried over which identifies the specific program or executable image. Here are some examples of behavior that is, and is not, allowed.

It doesn't matter whether the information is intentionally stored, or just "left over"; if it's about a specific program, it's not allowed:

• Allowed: when a program is run, its use of emulated instructions is noticed by the run-time dynamic optimizer, and these are replaced as it runs, during this run only, by native code.
• Not allowed: when the program is re-run, a disk cache is consulted to find out what instructions were replaced last time, and the replacement code is used instead of the original program.
• Not allowed: when the program is re-run, the replacement native instructions are still sitting in memory, and the replacement instructions are used instead of the original program.

If information is left over from a previous run that is not associated with a specific program, that is allowed:

• Allowed: a virtually-indexed branch predictor is trained during the reference run of 132.zeusmp2. When the second run of 132.zeusmp2 is begun, a portion of the branch predictor tables happen to still be in the state that they were in at the end of the previous run (i.e. some entries have not been re-used during runs of intervening programs).
• Not allowed: a branch predictor specifically identifies that certain virtual addresses belong to the executable for 132.zeusmp2, and on the second run of that executable it uses that knowledge.

Any form of RDO that uses memory about a specific program is forbidden:

• Allowed: while 132.zeusmp2 is running, the run-time dynamic optimizer notices that it seems to be doing a poor job of rearranging instructions for instruction cache packing today, and gives up for the duration of this run.
• Not allowed: the next time 132.zeusmp2 runs, the run-time dynamic optimizer remembers that it had difficulty last time and decides not to even try this time.
• Not allowed: the run-time dynamic optimizer recognizes that this new program is 132.zeusmp2 by the fact that it has the same filename, or has the same size, or has the same checksum, or contains the same symbols.

The system is allowed to respond to the currently running program, and to the overall workload:

• Allowed: the operating system notices that demand for big pages is intense for the currently running program, and takes measures to increase their supply.
• Not allowed: the operating system notices that the demand for big pages is intense for certain programs, and takes measures to supply big pages to those specific programs.
• Allowed: the operating system notices that the demand for big pages is intense today, and takes measures to increase the supply of them. This causes all but the first few SPEC OMP2012 benchmarks to run more quickly, as the BIGPAGE supply is improved.

4. Results Disclosure

SPEC requires a full disclosure of results and configuration details sufficient to reproduce the results. For results published on its web site, SPEC also requires that base results be published whenever peak results are published. If peak results are published outside of the SPEC web site (http://www.spec.org/omp2012/) in a publicly available medium, the tester must supply base results on request. Publication of results under non-disclosure or company internal use or company confidential are not "publicly" available.

A full disclosure of results must include:

1. The components of the disclosure page, as generated by the SPEC tools.
2. The tester's configuration file and any supplemental files needed to build the executables used to generate the results.
3. A flags definition disclosure.

A full disclosure of results must include sufficient information to allow a result to be independently reproduced. If a tester is aware that a configuration choice affects performance, then s/he must document it in the full disclosure.

Note: this rule is not meant to imply that the tester must describe irrelevant details or provide massively redundant information.

For information on how to publish a result on SPEC's web site, contact the SPEC office. Contact information is maintained at the SPEC web site, http://www.spec.org/.

4.1 Rules regarding availability dates and systems not yet shipped

If a tester publishes results for a hardware or software configuration that has not yet shipped,

1. The component suppliers must have firm plans to make production versions of all components generally available within 90 days of the first public release of the result (whether first published by the tester or by SPEC); and
2. The tester must specify the general availability dates that are planned.

Note 1: "Generally available" is defined in the SPEC High Performance Group Policy document, which can be found at http://www.spec.org/hpg/policy.html.

Note 2: It is acceptable to test larger configurations than customers are currently ordering, provided that the larger configurations can be ordered and the company is prepared to ship them.

4.1.1 Pre-production software can be used

A "pre-production", "alpha", "beta", or other pre-release version of a compiler (or other software) can be used in a test, provided that the performance-related features of the software are committed for inclusion in the final product.

The tester must practice due diligence to ensure that the tests do not use an uncommitted prototype with no particular shipment plans. An example of due diligence would be a memo from the compiler Project Leader which asserts that the tester's version accurately represents the planned product, and that the product will ship on date X.

The final, production version of all components must be generally available within 90 days after first public release of the result.

4.1.2 Software component names

When specifying a software component name in the results disclosure, the component name that should be used is the name that customers are expected to be able to use to order the component, as best as can be determined by the tester. It is understood that sometimes this may not be known with full accuracy; for example, the tester may believe that the component will be called "TurboUnix V5.1.1" and later find out that it has been renamed "TurboUnix V5.2", or even "Nirvana 1.0". In such cases, an editorial request can be made to update the result after publication.

Some testers may wish to also specify the exact identifier of the version actually used in the test (for example, "build 20070604"). Such additional identifiers may aid in later result reproduction, but are not required; the key point is to include the name that customers will be able to use to order the component.

4.1.3 Specifying dates

The configuration disclosure includes fields for both "Hardware Availability" and "Software Availability". In both cases, the date which must be used is the date of the component which is the last of the respective type to become generally available. The date is specified Mmm-YYYY as in the following examples: Jan-2007, Feb-2007. The Month is abbreviated to three letters with the first letter capitalized. A hyphen separates the Month and Year fields. The Year field is specified with four digits.

Since all components must be available within 90 days of the first public release of the result, the first day of the specified Month (and Year) must fall within this 90 day window.

4.1.4 If dates are not met

If a software or hardware date changes, but still falls within 90 days of first publication, a result page may be updated on request to SPEC.

If a software or hardware date changes to more than 90 days after first publication, the result is considered Non Compliant. For procedures regarding Non Compliant results, see the SPEC High Performance Group Policy Document, http://www.spec.org/hpg/policy.html.

4.1.5 Performance changes for pre-production systems

SPEC is aware that performance results for pre-production systems may sometimes be subject to change, for example when a last-minute bugfix reduces the final performance.

For results measured on pre-production systems, if the tester becomes aware of something that will reduce production system performance by more than 5% on an overall metric, the tester is required to republish the result, and the original result shall be considered Non Compliant.

Analogous rules apply to performance changes across post-production upgrades (Section 4.3.4).

4.2 Configuration Disclosure

The following sections describe the various elements that make up the disclosure of the system configuration tested. The SPEC tools allow setting this information in the configuration file, prior to starting the measurement (i.e. prior to the runspec command).

It is also acceptable to update the information after a measurement has been completed, by editing the rawfile. Rawfiles include a marker that separates the user-editable portion from the rest of the file.

# =============== do not edit below this point ===================

Edits are forbidden beyond that marker.

(There is information about rawfile updating in the rawformat section of the document utility.html.)

4.2.1 Identification of System, Manufacturer and Tester

Details are

1. Model Name
2. Test Date: Month, Year
3. Hardware Availability Date: Month, Year. If more than one date applies, use the latest one.
4. Software Availability Date: Month, Year. If more than one date applies, use the latest one.
5. Hardware Vendor
6. Test sponsor: the entity sponsoring the testing (defaults to hardware vendor).
7. Tester: the entity actually carrying out the tests (defaults to test sponsor).
8. OMP2012 license number of the test sponsor or the tester.

4.2.1.1 Identification of Equivalent Systems

SPEC recommends that measurements be done on the actual systems for which results are claimed. Nevertheless, SPEC recognizes that there is a cost of benchmarking, and that multiple publications from a single measurement may sometimes be appropriate. For example, two systems badged as "Model A" versus "Model B" may differ only in the badge itself; in this situation, differences are sometimes described as only "paint deep", and a tester may wish to perform only a single test (i.e. the runspec tool is invoked only once, and multiple rawfiles are prepared with differing system descriptions).

Although paint is usually not a performance-relevant difference, for other differences it can be difficult to draw a precise line as to when two similar systems should no longer be considered equivalent. For example, what if Model A and B come from different vendors? Use differing firmware, power supplies, or line voltage? Support additional types or numbers of disks, or other devices?

For SPEC OMP2012, a single measurement may be published as multiple equivalent results provided that all of the following requirements are met:

1. Performance differences from factors such as those listed in the paragraph above (paint, vendor, firmware, and so forth) are within normal run-to-run variation.

2. The CPU is the same.

3. The motherboards are the same:

   1. same motherboard manufacturer

   2. same electrical devices (for example, IO support chips, memory slots, PCI slots)

   3. same physical shape.

4. The memory systems are the same:

   1. same caches

   2. same number of memory modules

   3. memory modules are run at the same speed

   4. memory modules comply with same specifications, where applicable (for example, the same labels as determined by the JEDEC DDR3 DIMM Label Specification).

5. As tested, all hardware components are supported on both systems.

   For example, the Model A and Model B meet the requirements listed above, including a motherboard with the same number of DIMM slots. The Model A can be fully populated with 96 DIMMs. Due to space and thermal considerations, the Model B can only be half-populated; i.e. it is not supported with more than 48 DIMMs. If the actual sytem under test is the Model A, the tester must fill only the DIMM slots that are allowed to be filled for both systems.

6. Disclosures must reference each other, and must state which system was used for the actual measurement. For example:

   This result was measured on the Acme Model A. The Acme Model A and the Bugle Model B are equivalent.

7. No power measurement is made. A single power measurement may not be published as multiple equivalent systems.

When a single measurement is used for multiple systems, SPEC may ask for a review of the differences between the systems, may ask for substantiation of the requirements above, and/or may require that additional documentation be included in the publications.

4.2.2 Node Configuration

The system will consist of one node. Both the Hardware and the Software are described:

1. Model Name
2. Hardware Vendor
3. CPU Name: A manufacturer-determined processor formal name.
4. CPU Characteristics: Technical characteristics to help identify the processor.
1. This field must be used to disambiguate which processor is used, unless the CPU is already unambiguously designated by the combination of the fields "CPU Name", "CPU MHz", and "Level (n) Cache".
2. In addition, SPEC encourages use of this field to make it easier for the reader to identify a processor, even if the processor choice is not, technically, ambiguous.
3. SPEC does not require that OMP2012 results be published on the SPEC web site, although such publication is encouraged. For results that are published on its web site, SPEC is likely to use this field to note CPU technical characteristics that SPEC may deem useful for queries, and may adjust its contents from time to time.
4. Some processor differences may not be relevant to performance, such as differences in packaging, distribution channels, or CPU revision levels that affect a SPEC OMP2012 overall performance metric by less than 5%. In those cases, SPEC does not require disambiguation as to which processor was tested.

An example may help to clarify these four points:

For example, when first introduced, the TurboBlaster series is available with only one instruction set, and runs at speeds up to 2GHz. Later, a second instruction set (known as "Arch2") is introduced and older processors are commonly, but informally, referred to as having employed "Arch1", even though they were not sold with that term at the time. Chips with Arch2 are sold at speeds of 2GHz and higher. The manufacturer has chosen to call both Arch1 and Arch2 chips by the same formal chip name (TurboBlaster).

1. A 2.0GHz TurboBlaster result is published. Since the formal chip name is the same, and since both Arch1 and Arch2 are available at 2.0GHz, the CPU Characteristics field must be used to identify whether this is an Arch1 or Arch2 chip.
2. A 2.2GHz TurboBlaster result is published. In this case, there is technically no ambiguity, since all 2.2GHz results use Arch2. Nevertheless, the tester is encouraged to note that the chip uses Arch2, to help the reader disambiguate the processors.
3. As an aid to technical readers doing queries, SPEC may decide to adjust all the TurboBlaster results that have been posted on its website by adding either "Arch1" or "Arch2" to all posted results.
4. The 2.2GHz TurboBlaster is available in an OEM package and a Consumer package. These are highly similar, although the OEM version has additional testing features for use by OEMs. But these are both 2.2GHz TurboBlasters, with the same cache structure, same instruction set, and, within run-to-run variation, the same OMP2012 performance. In this case, it is not necessary to specify whether the OEM or Consumer version was tested.
5. CPU MHz: a numeric value expressed in megahertz. That is, do not say "1.0 GHz", say "1000". The value here is to be the speed at which the CPU is run, even if the chip itself is sold at a different clock rate. That is, if you "over-clock" or "under-clock" the part, disclose here the actual speed used.
6. CPU Maximum MHz: a numeric value expressed in megahertz. That is, do not say "1.0 GHz", say "1000". The value here is to be the maximum speed at which the CPU is run. This is typically what vendors call the maximum turbo frequency.
7. Level 1 (primary) Cache: Size, location, number of instances (e.g. "32 KB I + 64 KB D on chip per core").
8. Level 2 (secondary) Cache: Size, location, number of instances.
9. Level 3 (tertiary) Cache: Size, location, number of instances.
10. Other Cache: Size, location, number of instances.
11. Number of CPUs in the node It is assumed that processors can be described as containing one or more "chips", each of which contains some number of "cores", each of which can run some number of hardware "threads". Fields are provided in the results disclosure for each of these. If industry practice evolves such that these terms are no longer sufficient to describe processors, SPEC may adjust the field set.

The current fields are:

1. hw_ncores: number of processor cores enabled (total) during this test
2. hw_nchips: number of processor chips enabled during this test
3. hw_ncoresperchip: number of cores that are manufactured into a chip
4. hw_nthreadspercore: number of hardware threads enabled (per core) during this test

Regarding the fields in the above list that mention the word "enabled": if a node, chip, core, or thread is available for use during the test, then it must be counted. If one of these resources is disabled - for example by a firmware setting prior to boot - then it need not be counted, but the tester must exercise due diligence to ensure that disabled resources are truly disabled, and not silently giving help to the result.

Regarding the field (hw_ncoresperchip), the tester must count the cores irrespective of whether they are enabled.

Example: In the following tests, the SUT is a Turboblaster Model 32-64-256, which contains 32 chips. Each chip has 2 cores. Each core can run 4 hardware threads.

1. A 256-thread SPEC OMP2012 test uses all the available resources. It is reported as:

   hw_ncores:         64
   hw_nchips:         32
   hw_ncoresperchip:   2
   hw_nthreadspercore: 4
   				

2. The system is halted, and firmware commands are entered to enable 1 core per chip, and hardware threading is turned off. The system is booted, and a 32-thread test is run. The resources this time are:

   hw_ncores:         32
   hw_nchips:         32
   hw_ncoresperchip:   1
   hw_nthreadspercore: 1
   		

Note: if resources are disabled, the method(s) used for such disabling must be documented and supported.

12. Number of CPUs orderable. Specify the number of processors that can be ordered, using whatever units the customer would use when placing an order. If necessary, provide a mapping from that unit to the nodes/chips/cores units just above. For example:

    1 to 8 TurboCabinets. Each TurboCabinet contains 4 chips.
    		

13. Memory: Size in MB/GB. Performance-relevant information as to the memory configuration must be included, either in the field or in the notes section. If there is one and only one way to configure memory of the stated size, then no additional detail need be disclosed. But if a buyer of the system has choices to make, then the result page must document the choices that were made by the tester.
    For example, the tester may need to document number of memory carriers, size of DIMMs, banks, interleaving, access time, or even arrangement of modules: which sockets were used, which were left empty, which sockets had the bigger DIMMs.
    Exception: if the tester has evidence that a memory configuration choice does not affect performance, then SPEC does not require disclosure of the choice made by the tester.
    For example, if an 8GB system is known to perform identically whether configured with 8 x 1GB DIMMs or 4 x 2GB DIMMs, then SPEC does not require disclosure of which choice was made.
14. Disk Subsystem: Describes disks attached to processor nodes. File Server disks are documented as below. Details are Size (GB/TB), Type (SCSI, Fast SCSI etc.), and any other performance-relevant characteristics. The disk subsystem used for the SPEC OMP2012 run directories must be described. If other disks are also performance relevant, then they must also be described.
15. Other Hardware: Additional equipment added to improve performance (special disk controller, NVRAM file system accelerator etc.).

4.2.3 Software Configuration

This section describes the compiler invocation and running of the benchmarks. Details are:

1. Operating System: Name and Version


2. System State: On Linux systems with multiple run levels, the system state must be described by stating the run level and a very brief description of the meaning of that run level, for example:

   System State: Run level 4 (multi-user with display manager)

   On other systems:

   • If the system is installed and booted using default options, document the System State as "Default".
   • If the system is used in a non-default mode, document the system state using the vocabulary appropriate to that system (for example, "Safe Mode with Networking", "Single User Mode").

   Note: some Unix (and Unix-like) systems have deprecated the concept of "run levels", preferring other terminology for state description. In such cases, the system state field should use the vocabulary recommended by the operating system vendor.

Additional detail about system state may be added in free form notes.
3. File System Type used for the SPEC OMP2012 run directories.


4. Compilers:
   • C Compiler Name and Version
   • C++ Compiler Name and Version
   • Fortran Compiler(s) Name and Version


5. System Services: If performance relevant system services or daemons are shut down (e.g. remote management service, disk indexer / defragmenter, spyware defender, screen savers) these must be documented in the notes section. Incidental services that are not performance relevant may be shut down without being disclosed, such as the print service on a system with no printers attached. The tester remains responsible for the results being reproducible as described.

6. Scripted Installations and Pre-configured Software: In order to reduce the cost of benchmarking, test systems are sometimes installed using automatic scripting, or installed as preconfigured system images. A tester might use a set of scripts that configure the corporate-required customizations for IT Standards, or might install by copying a disk image that includes best practices of the performance community. SPEC understands that there is a cost to benchmarking, and does not forbid such installations, with the proviso that the tester is responsible to disclose how end users can achieve the claimed performance (using appropriate fields above).

   Example: the Corporate Standard Jumpstart Installation Script has 73 documented customizations and 278 undocumented customizations, 34 of which no one remembers. Of the various customizations, 17 are performance relevant for SPEC CPU2006 - and 4 of these are in the category "no one remembers". The tester is nevertheless responsible for finding and documenting all 17. Therefore to remove doubt, the tester prudently decides that it is less error-prone and more straightforward to simply start from customer media, rather than the Corporate Jumpstart.

7. Other Software: Additional software added to improve performance

4.2.4 Tuning Configuration

1. Base flags list.
2. Peak flags for each benchmark.
3. Portability flags used for any benchmark.
4. Base pointers: size of pointers in base.
1. "32-bit": if all benchmarks in base are compiled with switches that request only 32-bit pointers.
2. "64-bit": if all benchmarks in base are compiled with switches that request only 64-bit pointers.
3. "32/64-bit": if there is a mix of 32-bit and 64-bit
5. Peak pointers: size of pointers in peak.
6. System Tuning: System tuning must be documented. The free form notes must mention the tuning parameters (including BIOS settings) that have been applied. The definition of the parameters may be in the free form notes or in the flags file. Tuning parameters must also be documented and supported by the vendor.

7. Any additional notes such as listing any use of SPEC-approved alternate sources or tool changes.

8. If a change is planned for the spelling of a tuning string, both spellings should be documented in the notes section.

   For example, suppose the tester uses a pre-release compiler with:

   f90 -O4 --newcodegen --loopunroll:outerloop:alldisable
   	

   but the tester knows that the new code generator will be automatically applied in the final product, and that the spelling of the unroll switch will be simpler than the spelling used here. The recommended spelling for customers who wish to achieve the effect of the above command will be:

   f90 -O4 -no-outer-unroll
   	

   In this case, the flags report will include the actual spelling used by the tester, but a note should be added to document the spelling that will be recommended for customers.

9. Mapping of Threads to Cores on the System. Utilities used to perform the mapping should be described.
10. Any additional notes.

4.2.5 Description of Portability and Tuning Options ("Flags File")

SPEC OMP2012 provides benchmarks in source code form, which are compiled under control of SPEC's toolset. The SPEC tools automatically detect the use of compilation and linkage flags in the config file and document them in the benchmark configuration section of the final report. Both portability and optimization flags will be captured in the report subsection.

The SPEC tools require a flag description file which provides information about the syntax of the flags and their meanings. A result will be marked "invalid" unless it has an associated flag description file. A description of how to write one may be found at www.spec.org/omp2012/Docs .

The level of detail in the description of a flag is expected to be sufficient so that an interested technical reader can form a preliminary judgment of whether he or she would also want to apply the option.

• This requirement is phrased as a "preliminary judgment" because a complete judgment of a performance option often requires testing with the user's own application, to ensure that there are no unintended consequences.
• At minimum, if a flag has implications for safety, accuracy, or standards conformance, such implications must be disclosed.
• For example, one might write:
  When --algebraII is used, the compiler is allowed to use the rules of elementary algebra to simplify expressions and perform calculations in an order that it deems efficient. This flag allows the compiler to perform arithmetic in an order that may differ from the order indicated by programmer-supplied parentheses.
• The final sentence of the preceding paragraph is an example of a deviation from a standard which must be disclosed.

It is acceptable, and even common practice, for testers to build on each others' flags files, copying all or part of flags files posted by others into their own flags files; but doing so does not relieve an individual tester of the responsibility to ensure that the description is accurate.

Although these descriptions have historically been called "flags files", they must also include descriptions of other performance-relevant options that have been selected, including but not limited to environment variables, kernel options, file system tuning options, BIOS options, and options for any other performance-relevant software packages.

4.2.6 Power Measurement Devices

If an optional power measurement is reported, the following must included. More information on the fields can be found in the config file documentation.

• Power Analyzer
  • Hardware vendor: Company which manufactures and/or sales the power analyzer.
  • Model: The model name of the power analyzer type used for this benchmark run.
  • Connectivity: Which interface was used to connect the power analyzer to the PTDaemon host system and to read the power data, e.g. RS-232 (serial port), USB, GPIB etc.
  • Serial number: The serial number uniquely identifying the power analyzer used for this benchmark run, in order to help making sure that the power analyzer has been calibrated in the last 12 months.
  • Calibration of the Power Analyzer:
    • Institute
    • Accredited by
    • Calibration label
    • Date of calibration
  • Voltage range used for the measurement: Value of voltage range setting to which the power analyzer has been configured.
  • Electrical current range used for the measurement: Value of current range setting to which the power analyzer has been configured.
  • Input connection: Input connection used to connect the load, if several options are available, or "Default" if not.
  • Host system on which PTDaemon runs: The manufacturer and model number of the system connected to power analyzer and running the power daemon.
  • OS of host system: The name and the version of the operating system installed on the power daemon host system.
  • Version: The version of the power daemon program reading the analyzer data. This information is provided automatically by the benchmark software.
  • Setup description: Free format textual description of the device or devices measured by this power analyzer and the accompanying PTDaemon instance, e.g. "SUT Power Supplies 1 and 2". This is especially useful if more than one power analyzer is used.
• Temperature Sensor
  • Hardware vendor: Company which manufactures and/or sales the temperature sensor.
  • Model: The model name of the power analyzer type used for this benchmark run.
  • Driver version: The version number of the operating system driver used to control and read the temperature sensor
  • Connectivity: Which interface was used to read the temperature data from the sensor, e.g. RS-232 (serial port), USB etc.
  • Host system on which PTDaemon runs: The manufacturer and model number of the system connected to the temperature sensor and running the power daemon.
  • OS of host system: The name and the version of the operating system installed on the power daemon host system.
  • Version: The version of the power daemon program reading the sensor data. This information is provided automatically by the benchmark software.
  • Setup description: Free format textual description of the device or devices measured and the approximate location of this temperature sensor, e.g. "50 mm in front of SUT main airflow intake". This is especially useful if more than one temperature sensor is used.

The power analyzer needs to have been calibrated in the last 12 months. Auto-ranging is not allowed. The voltage and current range used for measurement must be reported. See the configuration file documentation for more details. Also, the ranges used by the analyzer must be reportable by the SPEC PTDaemon in order to ensure that an uncertainty calculation can be made.

4.2.7 Configuration Disclosure for User Built Systems

SPEC OMP2012 results are for systems, not just for chips: it is required that a user be able to obtain the system described in the result page and reproduce the result (within a small range for run-to-run variation).

Nevertheless, SPEC recognizes that chip and motherboard suppliers have a legitimate interest in OpenMP benchmarking. For those suppliers, the performance-relevant hardware components typically are the cpu chip, motherboard, and memory; but users would not be able to reproduce a result using only those three. To actually run the benchmarks, the user has to supply other components, such as a case, power supply, and disk; perhaps also a specialized CPU cooler, extra fans, a disk controller, graphics card, network adapter, BIOS, and configuration software.

Such systems are sometimes referred to as "white box", "home built", "kit built", or by various informal terms. For SPEC purposes, the key point is that the user has to do extra work in order to reproduce the performance of the tested components; therefore, this document refers to such systems as "user built".

For user built systems, the configuration disclosure must supply a parts list sufficient to reproduce the result. As of the listed availability dates in the disclosure, the user should be able to obtain the items described in the disclosure, spread them out on an anti-static work area, and, by following the instructions supplied with the components, plus any special instructions in the SPEC disclosure, build a working system that reproduces the result. It is acceptable to describe components using a generic name (e.g. "Any ATX case"), but the recipe must also give specific model names or part numbers that the user could order (e.g. "such as a Mimble Company ATX3 case").

Component settings that are listed in the disclosure must be within the supported ranges for those components. For example, if the memory timings are manipulated in the BIOS, the selected timings must be supported for the chosen type of memory.

Components for a user built system may be divided into two kinds: performance-relevant (for SPEC OMP2012), and non-performance-relevant. For example, SPEC OMP2012 benchmark scores are affected by memory speed, and motherboards often support more than one choice for memory; therefore, the choice of memory type is performance-relevant. By contrast, the motherboard needs to be mounted in a case. Which case is chosen in not normally performance-relevant; it simply has to be the correct size (e.g. ATX, microATX, etc).

• Performance-relevant components must be described in fields for "Configuration Disclosure" (see rules 4.2.2, and 4.2.4). These fields begin with hw_ or sw_ in the config file, as described in config.html (including hw_other and sw_other, which can be used for components not already covered by other fields). If more detail is needed beyond what will fit in the fields, add more information under the free-form notes.
• Components that are not performance-relevant are to be described in the free-form notes.

Example:

hw_cpu_name    = Frooble 1500 
hw_memory      = 2 GB (2x 1GB Mumble Inc Z12 DDR2 1066) 
sw_other       = SnailBios 17
notes_plat_000 = 
notes_plat_005 = The BIOS is the Mumble Inc SnailBios Version 17,
notes_plat_010 = which is required in order to set memory timings
notes_plat_015 = manually to DDR2-800 5-5-5-15.  The 2 DIMMs were
notes_plat_020 = configured in dual-channel mode. 
notes_plat_025 = 
notes_plat_030 = A standard ATX case is required, along with a 500W
notes_plat_035 = (minimum) ATX power supply [4-pin (+12V), 8-pin (+12V)
notes_plat_040 = and 24-pin are required].  An AGP or PCI graphics
notes_plat_045 = adapter is required in order to configure the system.
notes_plat_050 =
notes_plat_055 = The Frooble 1500 CPU chip is available in a retail box,
notes_plat_060 = part 12-34567, with appropriate heatsinks and fan assembly.  
notes_plat_065 =
notes_plat_070 = As tested, the system used a Mimble Company ATX3 case,
notes_plat_075 = a Frimble Ltd PS500 power supply, and a Frumble
notes_plat_080 = Corporation PCIe Z19 graphics adapter.
notes_plat_085 = 

Additional notes:

Note 1: Regarding graphics adapters:

• Sometimes a motherboard does not provide a graphics adapter. For many operating systems, in order to install the software, a graphics card must be added; for such cases where the use of the graphics adapter does not affect performance, it can be described in the free form notes, and does not need to be listed in the field "Other Hardware".
• If SPEC OMP2012 performance improves when the external adapter is added, it is, therefore, performance relevant: list the graphics adapter that was used under "Other Hardware".

Note 2: Regarding power modes: Sometimes CPU chips are capable of running with differing performance characteristics according to how much power the user would like to spend. If non-default power choices are made for a user built system, those choices must be documented in the notes section.

Note 3: Regarding cooling systems: Sometimes CPU chips are capable of running with degraded performance if the cooling system (fans, heatsinks, etc.) is inadequate. When describing user built systems, the notes section must describe how to provide cooling that allows the chip to achieve the measured performance.

4.3 Test Results Disclosure

The actual test results consist of the elapsed times and ratios for the individual benchmarks and the overall SPEC metric produced by running the benchmarks via the SPEC tools. The required use of the SPEC tools ensures that the results generated are based on benchmarks built, run, and validated according to the SPEC run rules.

4.3.1 OMP2012 Performance Metrics

Below is a list of the measurement components for each SPEC OMP2012 suite and metric:

• SPECompG_base2012 (Required Base result)
• SPECompG_peak2012 (Optional Peak result)
• SPECompG_2012 (Greater of Base and Peak result)

These are calculated as follows:

1. For the given OMP2012 suite, the elapsed time in seconds for each of its benchmark runs is reported.
2. The ratio of the reference system (Sun Fire X4140) time divided by the corresponding measured time is reported.
3. Separately for base and peak, the median of three runs of these ratios is reported per benchmark.
4. The "base" metric is the geometric mean of medians of the base ratios, and the "peak" metric is the geometric mean of medians of the peak ratios.

All runs of a specific benchmark when using the SPEC tools are required to have validated correctly. The benchmark executables must have been built according to the rules described in section 2 above.

4.3.2 OMP2012 Energy Metrics

Below is a list of the energy metrics for each SPEC OMP2012 suite:

• SPECompG_energy_base2012 (Optional Base result)
• SPECompG_energy_peak2012 (Optional Peak result)

These are calculated as follows:

1. For the given OMP2012 suite, the energy measurement for each of its benchmark runs is reported.
2. The ratio of the energy measurement divided by the corresponding reference system (Sun Fire X4140) energy is reported.
3. Separately for base and peak, the median of three runs of these ratios is reported per benchmark.
4. The "base" metric is the geometric mean of medians of the base ratios, and the "peak" metric is the geometric mean of medians of the peak ratios.

4.3.3 Metric Selection

Publication of energy and peak performance results are considered optional by SPEC, so the tester may choose to publish only base performance results. Since by definition base performance results adhere to all the rules that apply to peak performance results, the tester may choose to refer to these results by either the base or peak metric names (e.g. SPECompG_base2012 or SPECompG_peak2012) or the name SPECompG_2012 whose value is the greater of SPECompG_base2012 and SPECompG_peak2012.

It is permitted to publish base-only results. Alternatively, the use of the flag basepeak is permitted, as described in section 3.6.

4.3.4 Estimates are allowed

SPEC OMP2012 metrics may be estimated, with the exception of power measurements. All estimates must be clearly identified as such. It is acceptable to estimate a single metric (for example, SPECompG_base2012, or SPECompG_peak2012, or the elapsed seconds for 362.fma3d). Note that it is permitted to estimate a peak metric without being required to provide a corresponding estimate for base.

SPEC requires that every use of an estimated number be clearly marked with "est." or "estimated" next to each estimated number, rather than burying a footnote at the bottom of a page.

Model 1   SPECompG_base2012        50 est.
          SPECompG_peak2012        60 est.
Model 2   SPECompG_base2012        70 est.
          SPECompG_peak2012        80 est.

If estimates are used in graphs, the word "estimated" or "est." must be plainly visible within the graph, for example in the title, the scale, the legend, or next to each individual result that is estimated.

Note: the term "plainly visible" in this rule is not defined; it is intended as a call for responsible design of graphical elements. Nevertheless, for the sake of giving at least rough guidance, here are two examples of the right way and wrong way to mark estimated results in graphs:

• Acceptable: a 3 inch by 4 inch graph has 12 point (=1 pica) "est." markings directly above the top of every affected bar, using black type against a white background.
• Unacceptable: a 1 meter by 3 meter poster has 12 point "est." markings ambiguously placed, with light gray text on a dark gray background.

Licensees are encouraged to give a rationale or methodology for any estimates, together with other information that may help the reader assess the accuracy of the estimate. For example:

1. "This is a measured estimate: SPEC OMP2012 was run on pre-production hardware. Customer systems, planned for Q4, are expected to be similar."
2. "Performance estimates are modeled using the cycle simulator GrokSim Mark IV. It is likely that actual hardware, if built, would include significant differences."

Those who publish estimates are encouraged to publish actual SPEC OMP2012 metrics as soon as possible.

4.3.5 Performance changes for production systems

As mentioned previously, performance may sometimes change for pre-production systems; but this is also true of production systems (that is, systems that have already begun shipping). For example, a later revision to the firmware, or a mandatory OS bugfix, might reduce performance.

For production systems, if the tester becomes aware of something that reduces performance by more than 5% on an overall metric (for example, SPECompG_2012 or SPECompG_peak2012), the tester is encouraged but not required to republish the result. In such cases, the original result is not considered Non Compliant. The tester is also encouraged, but not required, to include a reference to the change that makes the results different (e.g. "with OS patch 20020604-02").

4.4 Required Disclosures

If a SPEC OMP2012 licensee publicly discloses an OMP2012 result (for example in a press release, academic paper, magazine article, or public web site), and does not clearly mark the result as an estimate, any SPEC member may request that the rawfile(s) from the run(s) be sent to SPEC. The rawfiles must be made available to all interested members no later than 10 working days after the request. The rawfile is expected to be complete, including configuration information (section 4.2 above).

A required disclosure is considered public information as soon as it is provided, including the configuration description.

For example, Company A claims a result of 1000 SPECompG_peak2012. A rawfile is requested, and supplied. Company B notices that the result was achieved by stringing together 50 chips in single-user mode. Company B is free to use this information in public (e.g. it could compare the Company A system vs. a Company B system that scores 999 using only 25 chips in multi-user mode).

Review of the result: Any SPEC member may request that a required disclosure be reviewed by the SPEC HPG subcommittee. At the conclusion of the review period, if the tester does not wish to have the result posted on the SPEC result pages, the result will not be posted. Nevertheless, as described above, the details of the disclosure are public information.

When public claims are made about OMP2012 results, whether by vendors or by academic researchers, SPEC reserves the right to take action if the rawfile is not made available, or shows different performance than the tester's claim, or has other rule violations.

4.5 Research and Academic usage of OMP2012

SPEC encourages use of the OMP2012 suite in academic and research environments. It is understood that experiments in such environments may be conducted in a less formal fashion than that demanded of testers who publish on the SPEC web site. For example, a research environment may use early prototype hardware that simply cannot be expected to stay up for the length of time required to meet the continuous run requirement (see section 3.5), or may use research compilers that are unsupported and are not generally available (see section 1).

Nevertheless, SPEC would like to encourage researchers to obey as many of the run rules as practical, even for informal research. SPEC respectfully suggests that following the rules will improve the clarity, reproducibility, and comparability of research results.

Where the rules cannot be followed, SPEC requires that the deviations from the rules be clearly disclosed, and that any SPEC metrics (such as SPECompG_2012) be clearly marked as estimated.

It is especially important to clearly distinguish results that do not comply with the run rules when the areas of non-compliance are major, such as not using the reference workload, or only being able to correctly validate a subset of the benchmarks.

4.6 Fair Use

Any public use of SPEC OMP2012 results must, at the time of publication, adhere to the then-currently-posted version of SPEC's Fair Use Rules, http://www.spec.org/fairuse.html.

If a competitive comparison uses SPEC OMP2012, it must use one or more of the following metrics:

1. The overall results: SPECompG_base2012, SPECompG_peak2012, SPECompG_2012;
2. Median run times of the individual benchmarks [see section 4.3.1];
3. Median individual benchmark SPEC ratios [see section 4.3.1];

The basis for comparison must be stated. Information from result pages may be used to define a basis for comparing a subset of systems, including but not limited to operating system version, cache size, memory size, compiler version, or compiler optimizations used.

All public statements regarding SPEC, its benchmarks, and especially results posted at www.spec.org, are required to be scrupulously correct as of the date listed in the public statement. However, there is no requirement to update public statements as new results are published. For example, if a web page says that the Turboblaster 1000 has "the best SPECompG_2012 result when compared versus all results published at www.spec.org through July 1, 2013", there is no requirement to change that web page if a better result is published on July 2.

Note: Regarding the use of non-median individual benchmark results: As described in section 4.3, each benchmark is run multiple times and the median is picked from the set of runs. Any result from such a set may be mentioned in a competitive comparison, provided that the median from the same set is also mentioned.

4.7 Submitting Results to SPEC

You have decided you want to not only run the benchmark, but to have your results published at SPEC. What do you do? First, be aware, that if you are not a member of SPEC HPG, there is a cost associated with publishing results at SPEC. Please check the SPEC website section submitting results, for more information about costs.

The following lists the steps to do when submitting results for review and publication.

• First, read the run rules. If you do not follow the run rules, your results may not be valid and be rejected.
• Next, create a configuration file. The document config.html contains information about how to create and change configuration files. A good starting place for a configuration file are the examples in the config subtree, or starting from previously submitted results which have been published at www.spec.org.
• Then after you get done getting things how you want them, you need to make a reportable run.
• After your reportable run is finished, and you are happy with the results, you can edit the documentation portion of the results (raw) file. You can use the rawformat command to help you verify your results and make changes to help you get your result ready for submission.
• Now you have gotten the result ready to go, you need to send it to SPEC. This is done by attaching the results file (.rsf) to an email and address it to subomp2012@spec.org. This is an automated system, and once it processes the submission, it will send you back a reply saying if it was accepted for review or needs more work.
• Results are reviewd on a two week cycle. You can see the schedule at the calendar.
• If you run into the need for support, please submit questions to omp2012support@spec.org.

5. Run Rule Exceptions

If for some reason, the test sponsor cannot run the benchmarks as specified in these rules, the test sponsor can seek SPEC approval for performance-neutral alternatives. No publication may be done without such approval. The SPEC High Performance Group (HPG) maintains a Policies and Procedures document that defines the procedures for such exceptions.

Copyright © 1999-2012 Standard Performance Evaluation Corporation
All Rights Reserved