SPEC CPU®2017 Technical Support

$Id: techsupport.html 6394 2019-11-20 13:45:02Z JohnHenning $ Latest: www.spec.org/cpu2017/Docs/

Contents

1. Introduction

2. Before contacting SPEC

3. Contact Information

4. Limitations on support

5. Checklist of information to include when contacting SPEC

6. Benchmark source code change requests

1. Introduction

SPEC CPU®2017 (a product of the SPEC® non-profit corporation (about SPEC)) is a source code benchmark. Using SPEC CPU 2017 requires access to a computer system running a supported operating system with the necessary compilers and tools to generate executable binaries for the system. The benchmarks require the use of compiler commands as well as other commands via a shell or command prompt window (aka "terminal window", "console", "CMD window", "terminal emulator", etc.)

SPEC can provide limited technical support for:

2. Before contacting SPEC

Do you have a problem with SPEC CPU 2017?
Try these, in this order:

  1. Update your copy with runcpu --update.
  2. Check the latest version of Frequently Asked Questions.
  3. Check the latest version of Known Problems.
  4. Check the src.alt list.    [What is a src.alt?]
  5. Contact SPEC.

3. Contact Information

SPEC CPU 2017 technical support can be reached by sending email to:

cpu2017support at spec.org

Additional/updated contact information can also be found at SPEC's primary Web Site: www.spec.org

4. Limitations on support

SPEC can provide only limited advice regarding:

In general, SPEC can provide technical assistance with its benchmarks, but SPEC does not have the ability to provide support on other vendor's products. Examples of support that SPEC can not provide include:

If known, SPEC will provide a pointer to the appropriate source of information. Otherwise for these and similar issues, it is suggested that the hardware or software vendor be contacted directly.

5. Checklist of information to include when contacting SPEC

When writing to SPEC to request help with a problem, please attach:

  1. Your config file
  2. The relevant .log file(s)

  3. The relevant .rsf file(s), if any .rsf files were produced.

    It is recommended that you attach the above 3 items as a compressed tarball. For example, if you are using my.cfg and the failure was seen during the 17th run, then you could create a file called problem.tar.xz using commands similar to these:
    Unix 
    cd $SPEC
spectar cvf - config/my.cfg result/CPU2017.017.log result/CPU2017.017*rsf | specxz > problem.tar.xz
    Windows 
    cd %SPEC%
spectar cvf problem.tar config\my.cfg result\CPU2017.017.log result\CPU2017.017*rsf
specxz problem.tar
    Both of the above create a file called problem.tar.xz
    In the Unix example, the output of spectar is piped to specxz on a single command line.
    In the Windows example, the same operations are done as 2 separate commands.
    In both cases, the file problem.tar.xz is created, which you can attach to your email.

  4. The revision of the benchmark suite that you're using - please say "runcpu -V"

  5. The specific nature of the problem and the symptoms.
    Please report the exact error message.
    If the message says "see file x/y/z.err", then please include the contents of x/y/z.err.
  6. Type of hardware/name of system being used.
  7. Name and version of the operating system.
  8. Name and version of the compiler.
  9. If a compilation issue:
  10. Has this issue been tried or seen on other systems?

Additional details related to the problem are also appreciated.

6. Benchmark source code change requests

Although the SPEC CPU 2017 benchmarks are very portable, new systems or language standard changes may require source code modifications. Under the run rules, no publication of results may be done using modified source code, unless SPEC has approved the change. This section describes how to go about making a request to SPEC for a source change.

By including all the relevant information, you will make it easier for SPEC to consider your request. Nevertheless, please note that SPEC may or may not grant your requested change. SPEC will consider a request for a source change in a similar manner to how it considers portability flags, weighing aspects such as performance neutrality, amount of code affected, and impact on the original intent of the program.

Proposals for changes to benchmark source code should include the information mentioned in the previous section, plus:

  1. A "context diff", typically via diff -u (or, if that is unavailable, diff -C). Specify the original file FIRST.

      diff -u  somefile.original.cxx  somefile.proposed.cxx

    If the changes are extensive, or if your diff utility does not know how to provide context-sensitive diffs, then attach the files. You can package them up and compress them by saying something like this:

      spectar cvf - newsrc/* | specxz > proposed.tar.xz
  2. A clear statement of what happens without the proposed change.
  3. A clear statement of whether a workaround exists. Are there any conditions under which the benchmark can be made to work without the proposed change?
  4. If possible, a specific reference to the relevant portion of the compiler language standard.
  5. If possible, results of testing for both correctness and performance when the same change is applied on some hw/sw platform other than the one where the problem was observed.
  6. If the problem has been reported to the compiler vendor, a reference to a specific bug number, case number, mail archive message number, web forum address, or compiler developer email address.
  7. If the problem has not been reported to the compiler vendor, a brief statement as to why not.
  8. If the problem needs to be solved by a certain date, please say so, and why.

Example change request: with [cross references] to above list

I would like to propose a change to benchmark 997.oldfort. Using SPEC CPU 2017 V1.0.3 [4], 997.oldfort fails to compile with error message:

  FATAL ERROR: foo.f, line 1814: DO loops must end with CONTINUE or END DO [5]

The errors can be seen in the attached tar file, which has my config file [1] and the log file [2]. It did not producs an .rsf [3] file.

The error occurs on the TurboBlaster Mark I system [6] using TurboBlasterUnix V1.0 [7] and TurboBlaster Fortran V1.0. [8] The compiler supports only two language dialects; neither one will compile the benchmark: --standard:Fortran2003 and --standard:Fortran2008 both fail. Lower optimization levels have no effect. [9]

The problem occurs only on the TurboBlaster. Other compilers (on other systems) issue a warning about the same line of source code, but not a fatal error. [10]

With the following change to 997.oldfort/src/foo.f, I can compile the benchmark: [11] 

   $ diff -u foo.f.orig foo.f
   --- foo.f.orig       Fri Jul 15 14:09:28 2011
   +++ foo.f    Sat Jul 29 10:06:34 2009
   @@ -1811,7 +1811,8 @@
          FFY(I) = ZERO
          DDT(I) = ZERO
          DDQ(I) = ZERO
   -  100 DDE(I) = ZERO
   +      DDE(I) = ZERO
   +  100 CONTINUE
          NYQBTTS = 20
          IF(QPM) NYQBTTS=21
    C
   $

Without the change, the compiler complains: [12]

FATAL ERROR: foo.f, line 1814: DO loops must end  either with CONTINUE or END DO 
   100 DDE(I) = ZERO
 --^

I am unable to workaround the problem on this system, because no other Fortran compiler is available, and TurboBlaster Fortran is quite insistent about this point. [13]

Although I do not have a copy of the standard, a Fortran 95 compiler on another system does warn about the same statement and specifically claims that it is nonstandard. [14] On that other system, the above source change was tested, and performance was the same (within run-to-run variation - much less than 1% different). [15] The problem has been reported to the compiler vendor, at www.turboblasterfortran.com/emailforum/msg00416.html, [16] and you can see in the replies attached to that message that the vendor seems quite proud of its assertion that it has a fully compliant Fortran2003 compiler with no support for older variations of the language.

It would be very much appreciated if you could consider this change request within the next month, so that I can publish my PhD dissertation before my parents arrive from overseas for a six-month visit. [18]

SPEC CPU®2017 Technical Support: Copyright © 2017-2019 Standard Performance Evaluation Corporation (SPEC®)