SPEC CPU®2017 Config Files

This document describes config files for SPEC CPU®2017, a product of the SPEC® non-profit corporation (about SPEC).

I. Config file basics

II. Config file options for runcpu

A. Precedence: config file vs. runcpu command line

B. Options

action allow_label_overrideNew backup_config basepeak bind check_version command_add_redirect copies current_rangeNew delay deletework difflines enable_monitorNew env_vars expand_notes expid fail fail_build fail_run feedback flagsurl force_monitorNew http_proxy http_timeout idle_current_rangeNew ignore_errors ignore_sigint info_wrap_columns iterations keeptmp labelNew line_width locking log_line_width log_timestamp mail_reports mailcompress mailmethod mailport mailserver mailto make make_no_clobber makeflags mean_anyway minimize_builddirs minimize_rundirs nobuild no_input_handler no_monitor notes_wrap_columns notes_wrap_indent output_format output_root parallel_test parallel_test_submit parallel_test_workloadsNew plain_train powerNew power_analyzerNew preenv rebuild reportable runlist save_build_filesNew section_specifier_fatal setprocgroup size src.alt strict_rundir_verify sysinfo_program table teeout temp_meterNew threadsNew train_single_threadNew train_with tune use_submit_for_compareNew use_submit_for_speed verbose verify_binariesNew voltage_rangeNew
Power measurement was introduced as an experimental feature in CPU 2017 v1.0. As of v1.1, more fields have been added, and the feature is now fully supported.

III. Config file options for specmake

IV. Config file options for the shell

V. Config file options for human readers

VI. Using Feedback Directed Optimization (FDO)

VII. The config file preprocessor

VIII. Output files - and how they relate to your config file

IX. Power Measurement

A. Getting started with power measurement

B. PTD examples

C. Descriptive fields for power and temperature

D. Current ranges

E. Power questions?

Appendices

A. Auxiliary benchmark sets

B. Troubleshooting

C. Obsolete/removed items

allow_extension_override company_name ext hw_cpu_char hw_cpu_mhz hw_cpu_ncoresperchip hw_fpu mach machine_name max_active_compares rate speed sw_auto_parallel test_date tester_name VENDOR

I. Config file basics

I.B. Benchmark selection

In a config file, you can reference: One or more individual benchmarks, such as 500.perlbench_r, or entire suites, using the Short Tags in the table below.

(Other benchmarks sets are available, but must be used with great caution. They are described in Appendix A.)

I.C. Five consumers

A config file has content for five (5) distinct consumers, as shown in the table.
To understand a config file, you must understand which consumer is addressed at any given point.
Column 3 below provides a few examples for each; click the roman numerals in column 2 for many more.

output_format = text,csv
tune          = base
reportable    = yes
runlist       = fpspeed 

runcpu --config=michael
runcpu --config=michael --output=text,csv --tune=base --reportable fpspeed 

fdo_pre0 = rm -Rf /tmp/manmohan/feedback
fdo_pre1 = mkdir /tmp/manmohan/feedback 

%if %{bits} == 64
%   define model  -m64
%else
%   define model  -m32
%endif
CC  = gcc %{model} 

I.D. Config file syntax

I.D.1. Sections and scope

A config file contains: a header section, named sections, and a HASH section.

Scope: Every line is considered to be within the scope of one of these three. Lines prior to the first section marker are in the scope of the header section. All other lines are either in the scope of the most recently preceding section marker or in the HASH section.

$ cat -n threeSections.cfg
 1  flagsurl             = $[top]/config/flags/gcc.xml
 2  iterations           = 1
 3  output_format        = text
 4  runlist              = 519.lbm_r
 5  size                 = test
 6  tune                 = peak
 7  fprate=peak:
 8     CC_VERSION_OPTION = -v
 9     CC                = gcc
10     OPTIMIZE          = -O2
11  519.lbm_r=peak:
12     OPTIMIZE          = -O3
13
14  __HASH__
15  519.lbm_r=peak=none:
16  # Last updated 2017-02-06 14:29:40
17  opthash=ff6059d6d9ec9577f7f49d05178c58688f31004089
18  baggage=
19  compiler_version=\
20  @eNo1jbEOgjAYhPc+RUcdaFEJIWwGHEhQjBLjRn7LLzapLWkL8
21  kpP9OCqkdX070hmtk0bTjMUspiuhQA9RFgfDki3brEkLdkCf0y
22  0NO36VHldDROLqTSzoNS2JfS5pT/DqUAH54cv4tAsjDEC6M9au
23  FuH/CZ+c5Q+pyRd+tUlX
24  compile_options=\
25  @eNp1T11PgzAUfe+vaPrOMjNfJGMJlLqhQBtLX3xptHYGBWoKM
26  vNp93Vjs3obadX2I+sHXZtD+0D3VXr9bX+8/InJBEMAeKBFZLq
27  KgWj06epUADLlCVqe+rFquJaKiHumJSaC1YWAuOAr/DPWvfu4I
28  mALF6zzeSpj96+RITFhJd3rG/dMaQTzEIJYV2T0DJl8RlGfl7Z
29  hjyd8vw+D9MirnY6z5LRW9NOC1yNkc/OfAJtxnuD
30  exehash=5290fe504238c6de1a13e275ab8ca11e035fbb4e7e
31
$
         

default=default=default:

default:

I.D.2. Comments

Comments begin with '#'. There are two types:

#> I didn't use the C++ beta because of Bob's big back-end bug.  

Both types of comments are ignored when processing the config file.

Full-line comments: If # is the first non-blank item on a line, then the whole line is a comment. Comment lines can be placed anywhere in a config file.

Trailing comments: If a line has non-blank items, you can (usually) add regular comments. You cannot write a protected trailing comment. If you try to use a protected comment after some other element, it is treated as a regular comment.

All comments below will be saved except the one that says 'NOT saved'.

                       # New optimizers.
default=base:          # Most optimizers go up to ten.
   OPTIMIZE   = -O11   #    These go to eleven.
                       #> This comment is NOT saved
   COPTIMIZE  = -std   #> This comment is saved

Not a comment: These instances of # do not start a comment:

• \#    To use a # without starting a comment, put a backslash in front of it:

  hw_model = Mr. Lee's \#1 Semi-Autonomous Unit
  

• notesN...#    Config file notes are printed in reports as entered, including any instances of #

  #notes100  = This note will not be printed, because # was the first item on the line.T
  notes110   = This note will be printed in reports.  All of it.  # Even this part.
  

I.D.3. Whitespace

Blank lines can be placed anywhere in a config file. They are ignored.

Spaces at the beginning of lines are ignored, with the exception that preprocessor directives always begin with a percent sign (%) in the first column. You can put spaces after the percent sign, if you wish (example).

Spaces within a line are usually ignored. Don't try to break up a section marker, and you can't break up a single word (say 'OPTIMIZE' not 'OPT I MIZE'). If multiple spaces separate line elements, it is as if there were only one space. Each of these have the same meaning:

   OPTIMIZE=-O2 -noalias
   OPTIMIZE   =  -O2      -noalias

One place where spaces are considered significant is in notes, where you can use spaces to line up your comments. Notes are printed in a fixed-width font.

Trailing spaces and tabs are ignored, unless they are preceded by a backslash. For example, if space.cfg contains:

$ cat trailing_space.cfg 
PATH1   = /path/without/any/trailer
FC      = $(PATH1)/f90
PATH2   = /path/with/trailing/space\
CC      = $(PATH2)/cc
$

then we can use fake to demonstrate the compile commands that would be generated (+ Unix commands grep, head, and cut to pick out one example).

$ cat trailing_space.sh 
runcpu -c space --fake 527.cam4 | grep without/ | head -1 | cut -b 1-80
runcpu -c space --fake 527.cam4 | grep trailing | head -1 | cut -b 1-80
$
$ ./trailing_space.sh 
/path/without/any/trailer/f90 -c -o ESMF_BaseMod.fppized.o -I. -Iinclude -Inetcd
/path/with/trailing/space /cc -c -o GPTLget_memusage.o -DSPEC -DSPEC_CPU -DNDEBU
$  (Notes about examples) 

Notice that the PATH2 trailing space is preserved

I.D.4. Quoting

If you use double (") or single (') quotes within a config file, runcpu leaves them alone. The assumption is that you put them there because one of the consumers (such as a shell) needs them. The quotes are not significant to runcpu but may be highly significant to the consumer. See the section on quote traps.

If you use a backslash (\) it is usually not significant. The exceptions are:

• \# prevents the pound sign from starting a comment.
• \space at the end of a line prevents the trailing space from being stripped.
• \$SYMBOL prevents runcpu variable substitution, typically because you are passing $SYMBOL to a shell

I.D.5. Line continuation

Many fields, including most reader fields, can be continued by adding a number:

sw_os1         = Turboblaster OS V1.0
sw_os2         = (Tested with Early Hardware Release 0.99
sw_os3         = and Patch 42.)
hw_disk105     = 42 TB on 6x 8 TB 10K RPM SAS Disks
hw_disk110     = arranged as 4x 2-way mirrors; plus
hw_disk115     = Turboblaster Disk Accelerator

The fields which cannot be continued are the ones that are expecting a simple integer, such as hw_nchips and license_num; and the ones that expect a date, such as hw_avail. You can pick your own style of numbering, as in the examples above. (Note: the stored results from your test always use three-digit numbers, and have slightly different syntax, as discussed in utility.html.)

Shell-style "here documents" with double angle brackets and a delimiter word (e.g. <<EOT) can be used to set multi-line values. Backslash-continued lines are also supported. For example:

$ cat continued_lines.cfg
expand_notes   = 1
output_format  = text
output_root    = /tmp/fake_lines
runlist        = 519.lbm_r

here_continued = <<EOT
      + This is +
       + a test  +
EOT

backslash_continued = + So is +\
 + this  +

notes1 = ${here_continued}
notes2 = ${backslash_continued}
$ cat continued_lines.sh 
runcpu --config=continued_lines --fakereport | grep txt
grep '+' /tmp/fake_lines/result/CPU2017*txt
$ ./continued_lines.sh 
    format: Text -> /tmp/fake_lines/result/CPU2017.001.fprate.refrate.txt
           + This is +
           + a test  +
     + So is +
     + this  +
$   (Notes about examples) 

I.D.6. Included files

You can include other files in your config file using include:
Multiple files may be included.
Included files may use macros (and you can use configpp to check the effect).
Included files may write to arbitrary sections, including (effectively) the header section.

Example: a config file is developed on one system, and applied on a different System Under Test (SUT). The compilers are, of course, installed on the development system. They might not be present on the SUT.

$ cat -n include.cfg
 1  iterations    = 1
 2  output_format = text
 3  output_root   = /tmp/example
 4  runlist       = 519.lbm_r
 5  size          = test
 6  include: SUT.inc
 7  default:
 8    CC                = gcc
 9    CC_VERSION_OPTION = -V
10    sw_compiler001    = C/C++/Fortran: Version 6.2.0 of GCC
11    sw_compiler002    = the GNU Compiler Collection
12    sw_avail          = Aug-2016
$

$ cp Turboblaster.inc SUT.inc
$ cat Turboblaster.inc 
default:
   hw_model  = SuperHero IV
   hw_avail  = Feb-2018
   hw_vendor = Turboblaster
$

$ cat include.sh 
runcpu --config=include | grep txt
grep avail /tmp/example/result/*txt
$ ./include.sh 
format: Text -> /tmp/example/result/CPU2017.001.fprate.test.txt
Test sponsor: Turboblaster      Hardware availability: Feb-2018
Tested by:    Turboblaster      Software availability: Aug-2016
$ (Notes about examples) 

I.D.7. Section markers and Named sections

A named section begins with a section marker and continues until the next section marker or the HASH section is reached.
Named sections can be entered in any order.
Section markers can be repeated. Material from repeated sections is automatically consolidated.

A section marker is a one- to three-part string of the form:

   benchmark[,...]=tuning[,...]=label[,...]:

The three parts of a section marker are called the section specifiers, with allowed values:

Trailing default section specifiers may be omitted from a section marker. In the pairs below, in each case, the second line is equivalent to the first:

intrate=default=default:     628.pop2_s=base=default:      default=default=default:
intrate:                     628.pop2_s=base:              default:

I.E. Section merging and precedence rules

By constructing section markers, you determine how you would like your options applied. Benchmarks are built according to instructions in the sections that they match, subject to rules for combining sections and resolving conflicts among them. Sections are combined using these rules.
Click to go to an example.

• For the benchmark specifier, the precedence is:

  highest   named benchmark(s)
            suite name
  lowest    default 

  (See also Appendix A)

• For the tuning specifier, base or peak has higher precedence than default.
• For the label specifier, any named label is at a higher precedence level than default.
• Combining sections:
  1. Combine sections that apply to a benchmark, if there is no conflict among them.

  2. If sections conflict with each other, the order of precedence is:

     highest     benchmark
                 tuning
     lowest      label 

• Section order:

  • For sections at differing precedence levels, order does not matter.

  • If a section occurs more than once, the settings are combined. If there are conflicts, the last instance wins.

  highest   named benchmark(s)
            suite name
  lowest    default

$ cat -n precedence_example1.cfg
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest
     5  runlist              = 519.lbm_r,619.lbm_s,505.mcf_r
     6  size                 = test
     7  default:
     8     CC_VERSION_OPTION = -v
     9     CC                = gcc
    10  519.lbm_r:
    11     OPTIMIZE          = -O3
    12  fpspeed:
    13     OPTIMIZE          = -O2
    14  fprate:
    15     OPTIMIZE          = -O1
    16  default:
    17     OPTIMIZE          = -O0
$ cat -n precedence_example1.sh 
     1  runcpu --config=precedence_example1 | grep txt
     2  cd /tmp/ptest/result
     3  grep 'O[0-9]' *txt
$ ./precedence_example1.sh 
    format: Text -> /tmp/ptest/result/CPU2017.001.fprate.test.txt
    format: Text -> /tmp/ptest/result/CPU2017.001.fpspeed.test.txt
    format: Text -> /tmp/ptest/result/CPU2017.001.intrate.test.txt
CPU2017.001.fprate.test.txt: 519.lbm_r: -O3
CPU2017.001.fpspeed.test.txt: 619.lbm_s: -O2
CPU2017.001.intrate.test.txt: 505.mcf_r: -O0
$ (Notes about examples) 

$ cat -n precedence_example2.cfg 
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest2
     5  runlist              = 519.lbm_r
     6  size                 = test
     7  default=default:
     8     CC_VERSION_OPTION = -v
     9     CC                = gcc
    10     OPTIMIZE          = -O0
    11  default=base:
    12     OPTIMIZE          = -O1
    13  default=peak:
    14     OPTIMIZE          = -O3
$ cat precedence_example2.sh 
runcpu --config=precedence_example2 --tune=base | grep txt
runcpu --config=precedence_example2 --tune=peak | grep txt
cd /tmp/ptest2/result
grep 'O[0-9]' *txt
$ ./precedence_example2.sh 
    format: Text -> /tmp/ptest2/result/CPU2017.001.fprate.test.txt
    format: Text -> /tmp/ptest2/result/CPU2017.002.fprate.test.txt
CPU2017.001.fprate.test.txt: 519.lbm_r: -O1
CPU2017.002.fprate.test.txt: 519.lbm_r: -O3
$  (Notes about examples) 

$ cat -n precedence_example3.cfg 
 1  runlist              = 619.lbm_s
 2  size                 = test
 3  fpspeed=base=default:
 4     CC_VERSION_OPTION = -v
 5     CC                = gcc
 6     OPTIMIZE          = -O0
 7  fpspeed=base=OhTwo:
 8     OPTIMIZE          = -O2
$ cat precedence_example3.sh 
runcpu --config=precedence_example3 --fake --label=OhTwo | grep lbm.c
$ ./precedence_example3.sh 
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DLARGE_WORKLOAD  -O2 lbm.c
$  (Notes about examples) 

$ cat -n precedence_example4.cfg 
  1  label                = wall
  2  runlist              = 619.lbm_s
  3  tune                 = peak
  4  default:
  5     CC_VERSION_OPTION = -v
  6     CC                = gcc
  7  fpspeed:
  8     OPTIMIZE          = -O1
  9  default=peak:
 10     COPTIMIZE         = -ftree-vectorize
 11  default=default=wall:
 12     EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example4.sh  
runcpu --config=precedence_example4 --fake | grep lbm.c
$ ./precedence_example4.sh 
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DLARGE_WORKLOAD
     -O1 -ftree-vectorize           -Wall     lbm.c
$  (Notes about examples) 

highest     benchmark
            tuning
lowest      label 

$ cat -n precedence_example5.cfg 
  1  flagsurl             = $[top]/config/flags/gcc.xml
  2  iterations           = 1
  3  output_format        = text
  4  output_root          = /tmp/ptest
  5  size                 = test
  6  default:
  7     CC_VERSION_OPTION = -v
  8     CC                = gcc
  9  fpspeed:
 10     OPTIMIZE          = -O3
 11  default=peak:
 12     OPTIMIZE          = -O2
 13  default=default=wall:
 14     OPTIMIZE          = -O1
 15  default:
 16     OPTIMIZE          = -O0
$ cat precedence_example5.sh 
runcpu --fake --config=precedence_example5 --label=wall 519.lbm          | grep lbm.c
runcpu --fake --config=precedence_example5 --label=wall -T peak  519.lbm | grep lbm.c
runcpu --fake --config=precedence_example5 --label=wall -T peak  619.lbm | grep lbm.c
$ ./precedence_example5.sh 
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP  -O1    lbm.c
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP  -O2    lbm.c
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DLARGE_WORKLOAD  -O3               lbm.c
$  (Notes about examples) 

$ diff --side-by-side precedence_example6a.cfg precedence_example6b.cfg
iterations           = 1                            iterations           = 1
output_format        = text                         output_format        = text
output_root          = /tmp/ptest                   output_root          = /tmp/ptest
runlist              = 519.lbm_r                    runlist              = 519.lbm_r
size                 = test                         size                 = test
default:                                            default:
   CC_VERSION_OPTION = -v                              CC_VERSION_OPTION = -v
   CC                = gcc                             CC                = gcc
519.lbm_r:                                        <
   OPTIMIZE          = -O3                        <
fprate:                                           <
   OPTIMIZE          = -O1                        <
default:                                            default:
   OPTIMIZE          = -O0                             OPTIMIZE          = -O0
                                                  > fprate:
                                                  >    OPTIMIZE          = -O1
                                                  > 519.lbm_r:
                                                  >    OPTIMIZE          = -O3
$ cat precedence_example6.sh
runcpu --fake --config=precedence_example6a | grep lbm.c
runcpu --fake --config=precedence_example6b | grep lbm.c
$ chmod +x precedence_example6.sh
$ ./precedence_example6.sh 
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP  -O3  lbm.c
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP  -O3  lbm.c
$  (Notes about examples) 

 7  fpspeed:
12  fpspeed=default:
17  fpspeed=default=default:

$ cat -n precedence_example7.cfg
  1  label                = wall
  2  runlist              = 619.lbm_s
  3  tune                 = peak
  4  default:
  5     CC_VERSION_OPTION = -v
  6     CC                = gcc
  7  fpspeed:
  8     OPTIMIZE          = -O1
  9     EXTRA_CFLAGS      = -finline-functions
 10  intrate:
 11     OPTIMIZE          = -O0
 12  fpspeed=default:
 13     OPTIMIZE          = -O2
 14     COPTIMIZE         = -ftree-vectorize
 15  intrate=peak:
 16     OPTIMIZE          = -O0
 17  fpspeed=default=default:
 18     OPTIMIZE          = -O3
 19     EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example7.sh 
runcpu --config=precedence_example7 --fake | grep lbm.c
$ ./precedence_example7.sh 
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DLARGE_WORKLOAD
    -O3 -ftree-vectorize  -finline-functions  -Wall   lbm.c
$  (Notes about examples) 

I.F. Variables and Variable Substitution

I.F.1. Defining variables

You can create your own runcpu variables using a line of the form

name = value

The name may contain only letters, digits, and underscores (a hyphen is NOT allowed).
Start with a letter.
You may indent your definitions if you wish (see: whitespace)

Exception: preprocessor macros are different on all of the above. Hyphens are allowed, use

%define name value

and the % must appear in column 1. You can indent after the % if you wish.

Conventions: Although not required, certain conventions are usually followed:

1. lower case is typically used for macros. When combining words, CamelCase may be useful.
2. lower case is also typically used for runcpu variables, combining words with underscores
3. CAPITALS are typically used when creating make variables
4. CAPITALS are also typically used for shell or environment variables.

Examples of the above (in the same order)

%ifndef processorNumaControl
%   define processorNumaControl firstTouch # macro
%endif
default:
    my_submit_cmd       = numactl -C $BIND # runcpu variable
    MYTUNE              = -O2 --math=SIMD  # make variable
    ENV_LD_LIBRARY_PATH = /opt/lib         # environment variable

The remainder of this section I.F is about runcpu variables -- the $[square] and ${curly} rows from the table at the top.

I.F.2. $[square] substitution at startup

Immediately after preprocessing, variables that are delimited by $[square brackets] are substituted.
Any value set in the config file can be substituted, provided that is visible in the current scope.
You can access the value of additional variables that you may have created.
See the list of useful variables below.
Perhaps the most useful is $[top] for the top of the SPEC CPU 2017 tree, often found in contexts such as:

    flagsurl01          = $[top]/config/compiler.xml
    flagsurl02          = $[top]/config/platform.xml
    EXTRA_LIBS          = $[top]/mylibs
    preENV_LIBRARY_PATH = $[top]/lib64:$[top]/lib

Traps for the unwary: In some cases it may be obvious to the human which value to use, but the tools aren't as smart as you.

1. Timing: You cannot substitute variables that do not exist, or whose values aren't known when the config file is first read. For example, the label is not sorted out until later. Values that pertain to individual benchmarks (such as the current iteration number or the name of the benchmark being run) are not set until much later, when benchmarks are actually being run.
2. Command line: If a value is mentioned on both the runcpu command line and in the config file, the substitution might not do what you expect.

3. output_root:
   You cannot set an output_root that depends on a runcpu variable.
   You can set one that uses a macro:

   output_root=$[top]/my/path      # wrong
   output_root=${top}/my/path      # wrong
   output_root=%{ENV_SPEC}/my/path # right

Square substitution is done early. That comes in handy if you need a variable right away, for example, in order to use it with preENV.

$ cat EarlySub.cfg 
SW_DIR                 = /opt/path/to/compilers
preENV                 = 1
preENV_LD_LIBRARY_PATH = $[SW_DIR]/lib

$ cat EarlySub.sh
runcpu --config=EarlySub --fake 519.lbm_r | grep '^LD' | uniq
$ ./EarlySub.sh 
LD_LIBRARY_PATH=/opt/path/to/compilers/lib
$  (Notes about examples) 

The example below uses variables defined in several named sections. The sections are delimited by section markers default: (line 8), default=base: (line 11), and default=peak: (line 15).

$ cat -n square.cfg
     1  expand_notes         = 1
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/square
     5  runlist              = 519.lbm_r
     6  size                 = test
     7  tune                 = base,peak
     8  default:
     9     CC                = gcc
    10     CC_VERSION_OPTION = -v
    11  default=base:
    12     the_system        = STAR
    13     OPTIMIZE          = -O1
    14     notes_base_100    = base tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    15  default=peak:
    16     OPTIMIZE          = -O2
    17     notes_peak_100    = peak tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    18
$ cat square.sh 
runcpu --config=square | grep txt
grep tuning /tmp/square/result/CPU2017.001.fprate.test.txt
$ ./square.sh 
    format: Text -> /tmp/square/result/CPU2017.001.fprate.test.txt
     base tuning uses 'gcc' '-O1' on system 'STAR'
     peak tuning uses 'gcc' '-O2' on system ''
$  (Notes about examples) 

Note that line 14 finds all three variables that it is looking for, but line 17 does not. If it is not clear why this happens, please see the descriptions of named sections and precedence above.

I.F.3. Useful $[square] variables

Useful $[variables] include:

You can access the initial value of most options that you can enter into a config file, including:

I.F.4. ${curly} substitution (interpolation) during a run
+ $unbracketed substitution

Runcpu uses perl interpolation.
Only scalars (not: perl arrays and hashes) can be interpolated.

Example: on the notes100 line, you could optionally add say either ${lognum} or $lognum, but don't try taking the curly brackets away from ${size}.

$ cat just.cfg 
expand_notes      = 1
notes100          = Just ${size}ing, in run $lognum
output_root       = /tmp/just
runlist           = 505.mcf_r
size              = test
CC                = gcc
CC_VERSION_OPTION = -v
$
$ cat just.sh
runcpu -c just | grep txt
grep Just /tmp/just/result/CPU2017.001.intrate.test.txt
$
$ ./just.sh 
    format: Text -> /tmp/just/result/CPU2017.001.intrate.test.txt
     Just testing, in run 001
$  (Notes about examples) 

Traps for the unwary

1. Try not to confuse ${curly} interpolation vs. %{curly} (macros)

2. Timing: Some variables are only defined at certain times, and a line that uses it might be interpolated at a different time. Therefore interpolation won't always do what you might wish. In particular, notes are not expanded in the context of a particular benchmark run, and therefore variables such as ${tune} are not useful within them.

3. output_root:
   You cannot set an output_root that depends on a runcpu variable.
   You can set one that uses a macro:

   output_root=$[top]/my/path      # wrong
   output_root=${top}/my/path      # wrong
   output_root=%{ENV_SPEC}/my/path # right

I.F.5. Useful ${curly} variables

These variables may be of interest:

For a complete list of the available variables relative to the current config file, set

expand_notes = 1
verbose      = 35 # or higher

Then, do a run that causes a command substitution to happen.
In the log, you will find many lines of the form:

 - Variables available for interpolation that have changed since the last list:
    (From config) $runmode = "rate"
    (From config) $size = "test"
 - Variables available for interpolation that have changed since the last list:
    (From config) $size = "train"

I.F.6. Unsetting a variable with "%undef%"

It is sometimes useful to undo the setting of a variable that would otherwise be included from another section. This can be accomplished using the special value %undef%. In the example, line 14 undefines COPTIMIZE when compiling peak:

 $ cat -n gnana.cfg 
 1  flagsurl             = $[top]/config/flags/gcc.xml
 2  iterations           = 1
 3  output_format        = text
 4  output_root          = /tmp/undef
 5  runlist              = 519.lbm_r
 6  size                 = test
 7  tune                 = base,peak
 8  default:
 9     CC_VERSION_OPTION = -v
10     CC                = gcc
11     OPTIMIZE          = -O2
12     COPTIMIZE         = -fno-tree-pre
13  519.lbm_r=peak:
14     COPTIMIZE         = %undef%
15
$ runcpu --config=gnana | grep txt
    format: Text -> /tmp/undef/result/CPU2017.001.fprate.test.txt
$ cd /tmp/undef/benchspec/CPU/519.lbm_r/build
$ grep OPTIMIZE build_base_none.0000/Makefile.spec 
COPTIMIZE        = -fno-tree-pre
OPTIMIZE         = -O2
$ grep OPTIMIZE build_peak_none.0000/Makefile.spec 
COPTIMIZE        =
OPTIMIZE         = -O2
$ (Notes about examples)   

Notice that in the build directory, COPTIMIZE is present for base and absent for peak.

I.F.7. Debug tips for runcpu variables

When debugging a config file that uses runcpu variables, try:

iterations       = 1
minimize_rundirs = 0
reportable       = 0
runlist          = (one or two benchmarks)
size             = test
verbose          = 40

Using --fake will probably be informative. Look inside the log for the (case-sensitive) word 'From'.

II. Config file options for runcpu

This section documents options that control the operation of runcpu itself.

II.A. Precedence: config file vs. runcpu command line

In the list that follows, some items are linked to the document Using SPEC CPU 2017 - the 'runcpu' Command because they can be specified either in a config file, or on the runcpu command line.

New with CPU 2017, If an option is specified in both places, the command line wins.

II.B. Options

In the table that follows, if an option is documented as accepting the values "no" and "yes", these may also be specified as "false" and "true", or as "0" and "1".

The "Use In" column indicates where the option can be used:

997.noisy=peak:
basepeak=yes

 bind = 0,1,2,3,4,5,6,7, 16,17,18,19,20,21,22,23     

bind = <<EOT
0,   1,   2,   3,   4,   5,   6,   7,
16, 17,  18,  19,  20,  21,  22,  23
EOT 

$ cat bindN.cfg 
copies      = 4
iterations  = 1
output_root = /tmp/submit
runlist     = 541.leela_r
size        = test
intrate:
   bind0    = assign_job cpu_id=11
   bind1    = assign_job cpu_id=13
   bind2    = assign_job cpu_id=17
   bind3    = assign_job cpu_id=19
   submit   = $BIND ${command}
$ runcpu --fake --config=bindN | grep '^assign' | cut -b 1-70
assign_job cpu_id=11 ../run_base_test_none.0000/leela_r_base.none test
assign_job cpu_id=13 ../run_base_test_none.0000/leela_r_base.none test
assign_job cpu_id=17 ../run_base_test_none.0000/leela_r_base.none test
assign_job cpu_id=19 ../run_base_test_none.0000/leela_r_base.none test
$(Notes about examples)    

submit = send_job job="${command}" cpu_id=$BIND

         send_job job="exe >out" cpu_id=n

         send_job job="exe" cpu_id=n >out

intrate=base:
   current_range = 3
557.xz_r=base:
   current_range = 4
intrate=peak:
   current_range = 5
557.xz_r=peak:
   current_range = 6

$ cat PerformabilityQOS.cfg
runlist     = 520.omnetpp_r
iterations  = 1
size        = test
tune        = peak
label       = srini
520.omnetpp_r=peak:
   CXX                 = g++
   CXX_VERSION_OPTION  = -v
   OPTIMIZE            = -O1
$ date
Mon Oct  3 17:49:46 PDT 2016
$ runcpu -c PerformabilityQOS | grep -i -e error: -e success
Build successes for intrate: 520.omnetpp_r(peak)
Success: 1x520.omnetpp_r
$ go 520.omnet exe
/export/home/rc2/benchspec/CPU/520.omnetpp_r/exe
$ ls -g | cut -b 27-88
 Oct  3 17:52 omnetpp_r_peak.srini
$(Notes about examples)    

$ ldd omnetpp_r_peak.srini
... version `GLIBCXX_3.4.20' not found (required by ./omnetpp_r_peak.srini)
... version `CXXABI_1.3.8' not found (required by ./omnetpp_r_peak.srini) 

$ diff -u PerformabilityQOS.cfg PerformabilityQOS.2.cfg 
--- PerformabilityQOS.cfg	2016-10-03 17:52:39.000000000 -0700
+++ PerformabilityQOS.2.cfg	2016-10-03 17:54:45.000000000 -0700
@@ -3,8 +3,10 @@
 size        = test
 tune        = peak
 label       = srini
+env_vars    = 1
 #
 520.omnetpp_r=peak:
+   ENV_LD_LIBRARY_PATH = %{ENV_SPEC}/libraries:%{ENV_LD_LIBRARY_PATH}
    CXX                 = g++
    CXX_VERSION_OPTION  = -v
    OPTIMIZE            = -O1
$ runcpu -c PerformabilityQOS | grep -i -e error: -e success
error: a total of 1 children finished with errors
$ runcpu -c PerformabilityQOS.2 | grep -i -e error: -e success
Success: 1x520.omnetpp_r
$ 

519.lbm_r=default:
#> I am posting this config file for use by others in the
#> company, but am forcing it to fail here because
#> I want to force users to review this section.
#>
#> Once you find your way here, you should test whether
#> bug report 234567 has been fixed, by using the first
#> line below.  If it has not been fixed, then use the
#> second.  In either case, you'll need to remove the
#> fail_build.
#>
#>   - Pney Guvaxre
#>     Boomtime, the 66th day of Confusion in the YOLD 3172

# OPTIMIZE = -Osuperduper
# OPTIMIZE = -Omiddling
fail_build = yes

     flagsurl1 = mycompiler.xml
     flagsurl2 = myplatform.xml 

http_proxy = http://webcache.tom.spokewrenchdad.com:8080

label=jun12.old.CC
label=jun12.new.CC
label=jun14-flagday
label=jun15-jeff.wants.yet.another.test

runcpu --label=yusoff [...]
ERROR: The label 'yusoff' defines no settings in the config file!

copies               = 3
output_format        = text
runlist              = 519.lbm_r
default:
   CC_VERSION_OPTION = -v
   CC                = gcc
default=base=OhZero:
   OPTIMIZE          = -O0
default=base=OhThree:
   OPTIMIZE          = -O3 

$ find . -type d -name '*OhZ*'
./benchspec/CPU/519.lbm_r/build/build_base_OhZero.0000
./benchspec/CPU/519.lbm_r/run/run_base_test_OhZero.0000
./benchspec/CPU/519.lbm_r/run/run_base_test_OhZero.0001
./benchspec/CPU/519.lbm_r/run/run_base_test_OhZero.0002
$ cd benchspec/CPU/519.lbm_r/build
$ grep OPTIMIZE */Makefile.spec
build_base_OhThree.0000/Makefile.spec:OPTIMIZE   = -O3
build_base_OhZero.0000/Makefile.spec:OPTIMIZE   = -O0
$ 

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail
   mail_reports=text

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail

runcpu --output_root=/Ouachita/Van --config=mine 520.omnetpp_r

Option	Use In	Default	Meaning
plain_train	H,N	yes	When set to yes (or true or 1), does not apply any submit commands to the feedback training run. It also causes the monitor_* hooks to be ignored for the feedback training run.
power	H	no	Enable/disable the optional power measurement mode. New Before changing this field, you should set: current range idle_current_range power analyzer temp meter voltage_range After those fields have been set, then: Power measurement can be enabled or disabled using either this config file option or by using the command line runcpu --power / --nopower options. In case of disagreement between the command line and the config file, the command line will win. See the section on Power Measurement, below.
power_analyzer	H	none	Network location (name and port) for the power analyzer to be used by the SPEC Power/Temperature Daemon (PTD). New Example: hex-analyzer001:8888 If you are using more than one power analyzer, separate them by commas. Example: hex-analyzer001:8888, hex-analyzer002:8888 The power_analyzer field tells the run time software where to find the analyzer. You also need to tell humans about your analyzer, using descriptive fields with matching {id} names, for example: hw_power_{id}_cal_date hw_power_{id}_connection The {id} contains only the numbers and letters from the analyzer location. Example: power_analyzer = hex-analyzer001:8888 hw_power_hexanalyzer0018888_cal_date = 3-Nov-2018 hw_power_hexanalyzer0018888_connection = Serial over USB See the section on Power Measurement, below.
preenv or pre_env	H	yes	Use preENV_ lines in the config file. When this option is set (the default), lines of the form preENV_<variable> = <value> will cause runcpu to set the specified environment variable to value and re-exec runcpu to perform the run. The restart is done in order to ensure that the entire run takes place with the new settings. You can set preENV_SOME_VARIABLE = value only in the header section or a section using one these benchmark specifiers: default: intrate: fprate: intspeed: fpspeed: Any attempt to use preENV_ in other sections is silently ignored. You cannot change OMP_NUM_THREADS using this feature. Use threads. Multiple preENV_ settings may appear in the config file. If you are looking for a way to affect the environment of an individual benchmark, try env_vars, not preenv. Example: Commonly, SPECspeed2017 Floating Point users will set a large stack for 627.cam4_s. You can do this by adding a line similar to this one: preENV_OMP_STACKSIZE = 120M to the header section (top) of your config file, or by adding a new section for fpspeed: fpspeed: preENV_OMP_STACKSIZE = 120M The exact size needed will vary depending on your operating system and compiler. You might need to adjust it. See also the Examples, in your installed SPEC CPU 2017 tree, in the config directory. There are traps for the unwary, as shown on the right (click to enlarge). Don't try putting preENV definitions in any named sections other than default: intrate: fprate: intspeed: or fpspeed: Don't use env_vars unless you want to affect only peak. It will work for base while you are doing your test runs; but when you switch to reportable, you will wonder why it stops working for all your base benchmarks.

$ cd $SPEC   # or on Microsoft Windows: cd %SPEC%
$ specxz -dc nnn.benchmark.FixMumble.tar.xz | spectar -xvf -
$ cat README.nnn.benchmark.src.alt.FixMumble.txt

$ cat testme.cfg
action               = buildsetup
runlist              = nnn.benchmark
tune                 = base
default:
   CC                = gcc
   CC_VERSION_OPTION = -v
nnn.benchmark=base=without:
   OPTIMIZE          = -O2
nnn.benchmark=base=with:
   OPTIMIZE          = -O2
   srcalt            = FixMumble

runcpu --label=without --config=testme
runcpu --label=with    --config=testme

sysinfo_program = 

sysinfo_program = <path_to_your_sysinfo_program>

temp_meter                                   = hex-analyzer001:8889
hw_temperature_hexanalyzer0018889_connection = USB
hw_temperature_hexanalyzer0018889_setup      = Attached directly to air inlet

III. Config file options for specmake

For SPEC CPU you do not write Makefiles. Instead, you set Make variables in the config file, which are sent to a SPEC-supplied copy of GNU Make, known as specmake. Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake. For example:

COMPILER_DIR = /usr/local/bin/
CC           = $(COMPILER_DIR)cc
CXX          = $(COMPILER_DIR)c++
FC           = $(COMPILER_DIR)f90

See below for more information on syntax of variables that you create and reference.

III.A. Commonly used Make variables

• Basic specmake variables are described in this section.
• Many more are described in the Make Variables document.
• The specmake utility can be run directly from the command line, as described in Utilities.

The following Make variables are frequently useful. When selecting where to put a flag, please bear in mind that the run rules require that portability flags must use PORTABILITY variables.

Example-PGI-linux-x86.cfg                CC_VERSION_OPTION  = -V
Example-gcc-linux-x86.cfg                CC_VERSION_OPTION  = -v
Example-intel-compiler-linux-rate.cfg    CC_VERSION_OPTION  = --version
Example-intel-compiler-windows-rate.cfg  CC_VERSION_OPTION  = -QV
Example-studio-solaris.cfg               CC_VERSION_OPTION  = -V
Example-xl-linux-ppc64le.cfg             CC_VERSION_OPTION  = -qversion=verbose

default:
   CC                 = /bin/xlc
   CC_VERSION_OPTION  = -qversion=verbose

   FC                 = /bin/gfortran
   FC_VERSION_OPTION  = -v

   CXX                = /turboblaster/c++
   CXX_VERSION_OPTION = --print-blaster
intrate=peak:
   CC                 = /bin/gcc
   CC_VERSION_OPTION  = -v
fpspeed=peak:
   FC                 = /bin/xlf
   FC_VERSION_OPTION  = -qversion=verbose

III.B. Using OpenMP and/orAutopar

New with CPU 2017: For fpspeed (SPECspeed 2017 Floating Point) and intspeed (SPECspeed2017 Integer), you may build using OpenMP and/or compiler auto-parallelization. This capability is much more likely to be useful for Floating Point, because:

• All SPECspeed Floating Point benchmarks include OpenMP directives
• For SPECspeed Integer, only the compression benchmark 657.xz_s includes OpenMP.
  (Note: although the benchmark 605.mcf_s source code has omp pragmas, these are suppressed at compile time due to validation reasons.)

III.B.1. Summary

specspeed:
   OPTIMIZE = -DSPEC_OPENMP Compiler switch for OpenMP 

specspeed:
   OPTIMIZE = -DSPEC_OPENMP -fopenmp    (GNU)
   OPTIMIZE = -DSPEC_OPENMP -qsmp=omp   (IBM XL)
   OPTIMIZE = -DSPEC_OPENMP -qopenmp    (Intel)
   OPTIMIZE = -DSPEC_OPENMP -xopenmp    (Oracle Studio)
   OPTIMIZE = -DSPEC_OPENMP -mp         (PGI) 

III.B.2. SPECrate: no OpenMP. No Autopar.

• For SPECrate, the rules do not allow use of OpenMP.
• For SPECrate, the rules do not allow use of compiler auto-parallelization.
• For SPECrate, no OpenMP directives are used.

III.B.3. SPECspeed: your choice

• For SPECspeed, you may optionally use OpenMP.
• For SPECspeed, you may optionally use compiler autopar.
• For SPECspeed, you may choose whether the OpenMP directives are visible to the compiler.

The config file fragment below demonstrates available options.

 1   intspeed:
 2      OPTIMIZE       = -DSPEC_SUPPRESS_OPENMP
 3   657.xz_s=peak:
 4      OPTIMIZE       = --openmp  -DSPEC_OPENMP
 5
 6   fpspeed:
 7      OPTIMIZE       = --openmp
 8      EXTRA_OPTIMIZE = -DSPEC_OPENMP
 9   603.bwaves_s=peak:
10      OPTIMIZE       = --autopar
11      EXTRA_OPTIMIZE = -DSPEC_SUPPRESS_OPENMP
12   619.lbm_s=peak:
13      OPTIMIZE       = --autopar
14      EXTRA_OPTIMIZE = -DSPEC_OPENMP

In the example above:

• Most of the SPECspeed integer benchmarks are built without any form of parallelization (line #2, above).
• Integer exception: For the peak build of 657.xz_s: OpenMP is used. (#4).
• Most of the SPECspeed floating point benchmarks are built with OpenMP (#7-8).
• FP Exception 1: For the peak build of 603.bwaves_s: OpenMP is not used; compiler autoparallelization is used (#10); no OpenMP directives are used. (#11)
• FP Exception 2: For the peak build of 619.lbm_s: compiler autopar is used (#13); the OpenMP directives are visible (#14). (It is conceivable that the compiler autopar facility might make use of information gained from those directives.)

III.B.4. Detail - conditions for enabling OpenMP

Having considered an example, let us now look at the conditions in detail. The form of the directives in the benchmarks is:

#if (defined(_OPENMP) || defined(SPEC_OPENMP)) && !defined(SPEC_SUPPRESS_OPENMP) && !defined(SPEC_AUTO_SUPPRESS_OPENMP)

How it works:

1. For reasons of how benchmark reporting is done, there are two duplicated controls that have precisely the same effect on sources:
   SPEC_SUPPRESS_OPENMP is available for you to set if you want to suppress OpenMP.
   SPEC_AUTO_SUPPRESS_OPENMP is used by runcpu when it automatically does so.
2. If you set -DSPEC_SUPPRESS_OPENMP for a SPECspeed benchmark, that removes the directives.
3. Or, if you set -DSPEC_OPENMP for a SPECspeed benchmark, that causes the directives to be visible to your compiler.

4. If you set neither one of them, then the visibility of the directives will depend upon whether the variable _OPENMP is set. Here, it is important to note that _OPENMP is defined by the OpenMP standard. It requires that if a compiler supports a preprocessor, it must set this variable. However, the SPEC CPU benchmarks do not use vendor-native Fortran preprocessors, because the Fortran standard does not define preprocessing. Instead, the preprocessing is done by filepp (specpp), which does not do anything to OpenMP variables unless explicitly told to do so. Thus the SPEC CPU toolset may behave differently than your vendor-native compiler.

   In short, you might find yourself scratching your head wondering why a Fortran OpenMP benchmark is so much faster when you compile it outside the SPEC CPU harness vs. inside.

   Recommendation: To avoid a sore scalp (and before blaming the SPEC CPU tools!), if you wish to enable OpenMP please make sure that you always set -DSPEC_OPENMP. If you do not, then directives might or might not be visible, depending on your compiler and depending on the language.

To make this more clear, assume that we are compiling these benchmark excerpts. (For simplicity, we leave aside the duplicated control.)

From 619.lbm_s (C)
   #if (defined(_OPENMP) || defined(SPEC_OPENMP)) && !defined(SPEC_SUPPRESS_OPENMP)
   #pragma omp parallel for
   #endif

From 603.bwaves_s (Fortran)
   #if (defined(_OPENMP) || defined (SPEC_OPENMP)) && !defined(SPEC_SUPPRESS_OPENMP)
   !$OMP PARALLEL DO PRIVATE(l,i,j,k)
   #endif

Further assume that we are using compilers that enable OpenMP by setting a flag called --openmp. The inclusion of the directives will be determined by the truth tables shown below. The important difference occurs at line 7: by the time the Fortran compiler is awakened, it is too late to enable the directives, because that decision was already made by specpp.

III.C. Creating your own Make variables

Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake.

Deprecated feature alert: Although it is also possible to pass information to specmake using curly brackets: ${COMPILER_DIR}, this is not recommended. Instead, you should consistently use curly brackets to address runcpu and round brackets to address specmake. It is possible that a future version of runcpu may insist on interpolating curly brackets itself, rather than allowing specmake to do so.

Example:

$ cat makevar.cfg 
action            = build
runlist           = 603.bwaves_s
default:
   DEBUG_SYMBOLS  = --debug:symbols=expanded_info_level:42
   EXTRA_FFLAGS   = $(GEE)
fpspeed=base:
   GEE            = -g $(DEBUG_SYMBOLS)
fpspeed=peak:
   GEE            = -g

$ cat makevar.sh
runcpu --config=makevar --fake --tune=base | grep COMP:
runcpu --config=makevar --fake --tune=peak | grep COMP:
$ ./makevar.sh 
COMP: "f90 -c -o options.o -g --debug:symbols=expanded_info_level:42 <source>"
COMP: "f90 -c -o options.o -g <source>"
$  (Notes about examples) 

The config file above creates two variables options (DEBUG_SYMBOLS and GEE). Both are passed to specmake, which interprets them. The results are shown above using the runcpu --fake option.

For an extensive example of variable substitution handled by specmake, see the SPEC CPU 2000 example at www.spec.org/cpu2000/docs/example-advanced.cfg. Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake.

III.D. The operator "+=" is available (but should be used with caution)

The operator "+=" adds to specmake variables. It may be convenient; it also may cause hard-to-diagnose bugs. Example:

$ cat tmp.cfg
action  = build
runlist = 519.lbm_r
tune    = peak
default:
   OPTIMIZE   = -O1
fprate=default:
   OPTIMIZE  += --unroll
default=peak:
   OPTIMIZE  += --inner_unroll
519.lbm_r=peak:
   OPTIMIZE  += --outer_unroll
default=default=breakfast:
   OPTIMIZE   = --jelly_roll

$ runcpu --fake --config=tmp | grep lbm.c
cc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -O1 --inner_unroll --unroll --outer_unroll lbm.c
$ (Notes about examples)

Note that the options accumulate.

Caution: although the += operator adds flexibility, it may introduce hard-to-predict behavior, depending on precedence of section specifiers, the order of your config file, and other features, such as include files. Instead of using '+=' try picking different make variables for different purposes. For an example of hard-to-predict behavior, what will happen if you add --label=breakfast to the above runcpu command? (Try it.)

Recommendations:
Avoid += to prevent surprises.
Keep it simple.
Pick different makevars for different purposes.
Create conventions for your config files and write them down, in config file comments.
If you must use += review its effects carefully (--fake is your friend).

III.E Using action=buildsetup to create a sandbox

When debugging a set of build options, it is often useful to create a "sandbox" - that is, a directory where you can play with the benchmark and its options. This example creates a build sandbox with action buildsetup.

$ cat sandbox.cfg 
action        = buildsetup
label         = fast
output_root   = /tmp/demo_buildsetup
runlist       = 519.lbm_r
tune          = peak
default=peak:
   OPTIMIZE   = --fast
$ cat sandbox.sh 
runcpu --config=sandbox | grep log
grep Makefile.spec /tmp/demo_buildsetup/result/CPU2017.001.log
$ ./sandbox.sh 
The log for this run is in /tmp/demo_buildsetup/result/CPU2017.001.log
Wrote to makefile '/tmp/demo_buildsetup/benchspec/CPU/519.lbm_r/build/build_peak_fast.0000/Makefile.spec':

The action causes a directory to be created
There, the Makefile can be examined, used in a dry run, or modified as part of a testing effort.

$ cd /tmp/demo_buildsetup/benchspec/CPU/519.lbm_r/build/build_peak_fast.0000/
$ grep OPTIMIZE Makefile.spec 
OPTIMIZE         = --fast
$ specmake --dry-run
cc -c -o lbm.o  -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP  --fast  lbm.c
cc -c -o main.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP  --fast  main.c
cc --fast lbm.o main.o -lm  -o lbm_r
$  (Notes about examples) 

See also the chapter on specmake in SPEC CPU 2017 Utilities and the sandbox examples in Avoiding runcpu.

III.F. About Automatic Rebuilds

The SPEC CPU tools try to keep config files and binaries synchronized with each other.(*)
Edits to a config file may cause binaries to be rebuilt, sometimes to the surprise(**) of testers.

Testing option sensitivity: The first thing that happens in a rebuild is to delete the old binary.
If that is a potential problem (perhaps it takes a long time to build), you can test whether a config file change will cause a rebuild:

1. Copy the config file to a test version (say, mycopy.cfg)
2. Edit the copy.
3. Remove any lines that change defaults for ignore_errors or verify_binaries
4. Use: runcpu --config=mycopy --nobuild --size=test --iterations=1 ...

The command:

• completes quickly (because --size=test and iterations=1)
• exits if binaries are not up to date, because --nobuild

No binaries are harmed.

IV. Config file options for the shell

Some options in your config file cause commands to be executed by your shell (/bin/sh) or by the Windows command interpreter (cmd.exe).

IV.A. \$SHELLVAR variable substitution

Substitution by the shell - or by the windows command interpreter - uses backslash dollar sign.
The backslash protects the variables from interpretation by runcpu.

$ cat tmp.cfg
expand_notes         = 1
iterations           = 1
runlist              = 519.lbm_r
size                 = test
tune                 = base,peak
default:
   CC                = gcc
   CC_VERSION_OPTION = -v
default=base:
   submit    = echo home=$HOME, spec=$SPEC   > /tmp/chan; ${command}
default=peak:
   submit    = echo home=\$HOME, spec=\$SPEC > /tmp/nui;  ${command}
$ runcpu --config=tmp | grep txt 
    format: Text -> /Users/chris/spec/cpu2017/result/CPU2017.697.fprate.test.txt
$ cd /tmp
$ cat chan nui
home=, spec=
home=/Users/chris, spec=/Users/chris/spec/cpu2017
$ (Notes about examples)   

IV.B. Shell Options

Warning: SPEC CPU config files can execute arbitrary shell commands.
Read a config file before using it.
Don't be root. Don't run as Administrator. Turn privileges off.

These options cause commands to be executed:

IV.C. Using submit

The config file feature submit allows you to distribute jobs across a multiprocessor system. This section provides examples to demonstrate how submit works with several other config file features. You might also want to search published results at www.spec.org/cpu2017 for systems that are similar to your system.

IV.C.1. Basic usage of submit

You can use your operating system's facilities that assign jobs to processors, such as dplace, pbind, procbind, prun, start/affinity, or taskset together with ${command} and $SPECCOPYNUM.

The example below runs 4 copies, sending each one to a different processor.

$ cat taskset.cfg 
copies   = 4
runlist  = 519.lbm_r
submit   = taskset -c $SPECCOPYNUM ${command}
$ cat taskset.sh
runcpu --fake --config=taskset | grep '^taskset' | cut -b 1-75
$ ./taskset.sh 
taskset -c 0 ../run_base_refrate_none.0000/lbm_r_base.none 3000 reference.d
taskset -c 1 ../run_base_refrate_none.0000/lbm_r_base.none 3000 reference.d
taskset -c 2 ../run_base_refrate_none.0000/lbm_r_base.none 3000 reference.d
taskset -c 3 ../run_base_refrate_none.0000/lbm_r_base.none 3000 reference.d
$  (Notes about examples) 

Notice that $SPECCOPYNUM acquires the values 0, 1, 2, 3 in the generated commands, thereby using a different taskset assignment for each.

A problem with the above example: you might not want to send copy #2 to processor #2 and copy #3 to processor #3.
Perhaps you have a system with processors that do not have contiguous ID numbers.
Perhaps you want to spread the work out across a system, or you want to alternate jobs in ping-pong fashion.
You can customize the destinations with a bind list.

A system has 512 virtual processors, 64 chips, 4 cores per chip, 2 threads per core.
We would like to run one copy per chip.
This system does processor binding using pbind -b processor -e command
The bind statement on lines 1 through 5 specifies one processor id from each chip.
Line 11 plugs them into $BIND

$ cat -n pbind.cfg
     1  bind = \
     2       0   8  16  24  32  40  48  56  64  72  80  88  96 104 112 120 \
     3     128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 \
     4     256 264 272 280 288 296 304 312 320 328 336 344 352 360 368 376 \
     5     384 392 400 408 416 424 432 440 448 456 464 472 480 488 496 504
     6  copies               = 64
     7  iterations           = 1
     8  output_root          = /tmp/pbind
     9  runlist              = 519.lbm_r
    10  size                 = test
    11  submit               = pbind -b $BIND -e ${command}
    12  verbose              = 40
    13  default:
    14     CC                = cc
    15     CC_VERSION_OPTION = -V
    16
$ cat pbind.sh
runcpu --config=pbind > /dev/null &  # put runcpu in background
sleep 15                             # let things get started
pbind -q > /tmp/pbind/bound          # query bindings
$ ./pbind.sh
$ (Notes about examples) 

The pbind.sh script starts the run, waits 15 seconds, and then checks the status with pbind -q.
Below, we verify that all 64 copies were bound: /tmp/pbind/bound has 64 lines. A few are shown.

$ wc -l /tmp/pbind/bound
      64 /tmp/pbind/bound
$ sort -nk8 /tmp/pbind/bound | head -6
pbind(1M): pid 22673 strongly bound to processor(s) 0.
pbind(1M): pid 22675 strongly bound to processor(s) 8.
pbind(1M): pid 22682 strongly bound to processor(s) 16.
pbind(1M): pid 22693 strongly bound to processor(s) 24.
pbind(1M): pid 22689 strongly bound to processor(s) 32.
pbind(1M): pid 22685 strongly bound to processor(s) 40.

IV.C.2. Script generation by submit

Although Example 2 sent 64 copies where we want them, we only know that because a separate process happened to be watching.
That's not good enough. It would be much better to always leave confirmation that submit does what is intended.
To do so, generate a small script for each copy.

This example uses echo to create a script called dobmk for each benchmark copy.
Line 12 writes a processor binding command to the script.
Line 13 appends the ${command} that executes the benchmark.
Line 14 actually runs it.

$ cat -n scriptGen.cfg 
     1  bind = \
     2       0   8  16  24  32  40  48  56  64  72  80  88  96 104 112 120 \
     3     128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 \
     4     256 264 272 280 288 296 304 312 320 328 336 344 352 360 368 376 \
     5     384 392 400 408 416 424 432 440 448 456 464 472 480 488 496 504
     6  command_add_redirect = yes
     7  copies               = 64
     8  iterations           = 1
     9  output_root          = /tmp/pbind
    10  runlist              = 519.lbm_r
    11  size                 = test
    12  submit0              = echo 'pbind -b $BIND \$\$ >> pbind.out' > dobmk
    13  submit2              = echo "${command}" >> dobmk
    14  submit4              = sh dobmk
    15  default:
    16     CC                = cc
    17     CC_VERSION_OPTION = -V
    18
$ cat scriptGen.sh
runcpu --config=scriptGen | grep copies
$ ./scriptGen.sh 
  Setting up 519.lbm_r test base none (64 copies): run_base_test_none.0000-0063
  Running 519.lbm_r test base none (64 copies) [2017-02-10 16:05:16]
$ (Notes about examples) 

A generated dobmk is below. Copy #42 gets bound to processor id 336, which is #42 in the bind list.
Notice that dobmk includes redirection operators, such as >lbm.out
The operators are present because config file line 6 sets command_add_redirect.

$ cd /tmp/pbind/benchspec/CPU/519.lbm_r/run
$ cat run_base_test_none.0042/dobmk 
pbind -b 336 $$ >> pbind.out
../run_base_test_none.0000/lbm_r_base.none 20 reference.dat 0 1
             100_100_130_cf_a.of > lbm.out 2>> lbm.err
$  [line wrap added for readability] 

All copies run the same benchmark binary, namely the one in directory 0000.
The name of the executable is lbm_r_base.none because label defaults to none.

Lastly, we can verify that the pbind command worked by looking at pbind.out:

$ cat run_base_test_none.0042/pbind.out 
pbind(1M): pid 24422 strongly bound to processor(s) 336.
$ 

The next example generates scripts for numactl.

On line 9, the config file creates a runcpu variable called numactlShow which contains within it the command to demonstrate processor assignment. It is written to dobmk along with the command to actually run the benchmark on lines 10-11. On line 12, dobmk is invoked.

$ cat -n numactl.cfg
     1  iterations              = 1
     2  output_root             = /tmp/numactl
     3  size                    = test
     4  tune                    = base
     5  verbose                 = 40
     6  default: # --------- submit stuff ----------------------------------------
     7     bind                 = 2,  3,  5,  7, 11, 13, 17, 19, 23, 29, 31
     8     command_add_redirect = yes
     9     numactlShow          = numactl --show | grep phys >> numactl-s.out 2>&1
    10     submit0              = echo "$[numactlShow]"        > dobmk
    11     submit3              = echo "${command}"           >> dobmk
    12     submit5              = numactl --physcpubind=$BIND sh dobmk
    13  default: #---------- compiler stuff --------------------------------------
    14     CC                   = gcc
    15     CC_VERSION_OPTION    = -v
    16     OPTIMIZE             = -O3
    17  intrate: #---------- suite stuff -----------------------------------------
    18     copies               = 11
    19
$ cat numactl.sh
runcpu --config=numactl --fake 557.xz_r | grep "The log for this run"
runcpu --config=numactl        557.xz_r | grep txt
$ ./numactl.sh
The log for this run is in /tmp/numactl/result/CPU2017.001.log
    format: Text -> /tmp/numactl/result/CPU2017.002.intrate.test.txt

When the compression program 557.xz_r runs its test workload, the benchmark binary is actually run 11 times, as it tests various types of compression. We can see the evidence of that if we look at a sample run directory. Notice that all 11 invocations of copy #9 used processor #9 from the bind list on line 7 of the config file.

$ cd /tmp/numactl/benchspec/CPU/557.xz_r/run
$ cd run_base_test_none.0009
$ cat numactl-s.out
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
physcpubind: 29
$  (Notes about examples) 

IV.C.3. Quoting and traps for the unwary

Submit Example 3 uses three types of quoting: backslashes, single quotes, and double quotes.

submit0 = echo 'pbind -b $BIND \$\$ >> pbind.out' > dobmk
submit2 = echo "${command}" >> dobmk
submit4 = sh dobmk

The details of the quoting may seem like more than what you want to know, but can be crucial if you develop or maintain submit options. Caution: here be traps. Quote carefully, then check whether it did what you think it did.

"" Double quotes are used on the submit2 line. They do not prevent interpretation of ${command} because runcpu pays no particular attention to double or single quotes. The command to run lbm_r_base.none is inserted with its arguments and device assignments, and the double quotes are still present when the echo executes, where they protect the device assignments.

For example, here is the echo command from copy #42 (how was this found?) [line wrap added for readability]:

echo "../run_base_test_none.0000/lbm_r_base.none 20 reference.dat 0
   1 100_100_130_cf_a.of > lbm.out 2>> lbm.err" >> dobmk;

TRAP: If you forget the double quotes, you effectively say:

"Please do an echo.
Send standard output to lbm.out.
Send standard error to lbm.err.
No, wait, send standard output to dobmk."

Your operating system does whatever it does with such an odd request; a likely result is that lbm.out is created by the echo, but it has zero bytes, because lbm_r_base.none is not told to write there.

\$\$ On the submit0 line, the designator for the current process is quoted using backslashes to prevent runcpu from trying to interpret it.
$BIND does not have a backslash, because we do want runcpu to substitute values from the bind list.
TRAP: If the submit0 uses $$ (without backslashes), runcpu substitutes its own process ID, and 64 dobmk scripts fight uselessly about the location of runcpu.

'' The submit0 line also uses single quotes, which are still present when the echo is done.
By then, $BIND has already been substituted, and \$\$ has become $$.
The single quotes prevent $$ substitution by the parent shell that runs echo.

For example, here is what runcpu generated for copy #42 (how was this found?)

echo 'pbind -b 336 $$ >> pbind.out' > dobmk

TRAP: If double quotes are used on submit0 instead of single quotes, an interesting bug happens: the parent shell pid is inserted, dobmk runs, and obediently binds its parent to a processor. It does not bind itself. ("Wait, wait", you say, "Bindings are inherited, so binding the parent should be fine, right?" Well, no. Bindings are inherited only for newly created processes. Binding your parent doesn't do a thing for yourself. So, you definitely want the $$ to become dobmk's PID, not its parent's.)

IV.C.4. Debug tips for submit

1. Start small. Try to debug your options for a run with, say, 3 copies before trying 2048.

2. When debugging submit options, increase the log verbose option.

3. Pick methods that leave evidence behind. If you generate little scripts (like dobmk above), they can be examined.

4. Capture standard output from your processor assignment command (e.g. pbind.out from line 12 of Example 3). (It would also be a very good idea to capture standard error.)

5. Use the specinvoke -n dry run option. On the system where Example 3 was run, a grep verifies that there are 64 generated pbind commands. Another grep picks out the one that belongs to copy #42, using head -43. Copy #42 is, of course, the 43rd copy to run, because copy #0 is the first.

   $ pwd
   /tmp/pbind/benchspec/CPU/519.lbm_r/run/run_base_test_none.0000
   $ specinvoke -n | grep -c pbind
   64
   $ specinvoke -n | grep pbind | head -43 | tail -1
   echo 'pbind -b 336 $$ >> pbind.out' > dobmk;
   echo "../run_base_test_none.0000/lbm_r_base.none 20 reference.dat 0
      1 100_100_130_cf_a.of > lbm.out 2>> lbm.err" >> dobmk;
   sh dobmk
   $ 

   (Because runcpu generates very wide lines, line wraps were inserted for readability.)

IV.C.5. Maintainability suggestions when using submit

Here are two maintainability suggestions for authors of config files that use submit.

• If you have followed the debug suggestions above, you possess evidence that your config file works as desired -- today. Preserve the evidence. For example, if you are generating little dobmk scripts, and they generate little tidbits on stdout or stderr to verify processor assignments, keep the tidbits. Resist the temptation to rip them out. They provide cheap insurance against mysterious failures, for example when a new operating system changes the environment.

• Although runcpu generates very wide lines, you should not. Your config file will be more maintainable if you limit your line width.

To limit line width, you can use any of the three continuation styles discussed at the top of this document.

All three styles are used in this config file. All have the same effect.

$ cat -n continued.cfg
     1  command_add_redirect = yes
     2  copies               = 8
     3  iterations           = 1
     4  output_root          = /tmp/submit
     5  runlist              = 505.mcf_r
     6  size                 = test
     7  use_submit_for_speed = yes
     8  default:
     9     CC                = cc
    10     CC_VERSION_OPTION = -V
    11  intrate=base=backslash:
    12     submit0 = echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out' > dobmk \
    13               echo "${command}" >> dobmk \
    14               sh dobmk
    15  intrate=base=fieldN:
    16     submit0 = echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out' > dobmk
    17     submit2 = echo "${command}" >> dobmk
    18     submit4 = sh dobmk
    19  intrate=base=heredoc:
    20     submit = <<EOT
    21               echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out' > dobmk
    22               echo "${command}" >> dobmk
    23               sh dobmk
    24  EOT

To run the config file, the script below uses three runcpu commands with three different --label switches. After the runs finish, it produces a list of dobmk files that were generated, and prints a sample of each type.

$ cat continued.sh
runcpu --config=continued --label=backslash | grep copies
runcpu --config=continued --label=fieldN    | grep copies
runcpu --config=continued --label=heredoc   | grep copies
#
cd /tmp/submit/benchspec/CPU/505.mcf_r/run
ls *backslash*/dobmk
ls *fieldN*/dobmk
ls *heredoc*/dobmk
#
for file in *0007/dobmk ; do
   echo ==== $file =====
   cat $file
done
$ ./continued.sh 
  Setting up 505.mcf_r test base backslash (8 copies): run_base_test_backslash.0000-0007
  Running 505.mcf_r test base backslash (8 copies) [2017-02-11 07:38:32]
  Setting up 505.mcf_r test base fieldN (8 copies): run_base_test_fieldN.0000-0007
  Running 505.mcf_r test base fieldN (8 copies) [2017-02-11 07:40:55]
  Setting up 505.mcf_r test base heredoc (8 copies): run_base_test_heredoc.0000-0007
  Running 505.mcf_r test base heredoc (8 copies) [2017-02-11 07:43:15]

run_base_test_backslash.0000/dobmk  run_base_test_backslash.0004/dobmk
run_base_test_backslash.0001/dobmk  run_base_test_backslash.0005/dobmk
run_base_test_backslash.0002/dobmk  run_base_test_backslash.0006/dobmk
run_base_test_backslash.0003/dobmk  run_base_test_backslash.0007/dobmk
run_base_test_fieldN.0000/dobmk  run_base_test_fieldN.0004/dobmk
run_base_test_fieldN.0001/dobmk  run_base_test_fieldN.0005/dobmk
run_base_test_fieldN.0002/dobmk  run_base_test_fieldN.0006/dobmk
run_base_test_fieldN.0003/dobmk  run_base_test_fieldN.0007/dobmk
run_base_test_heredoc.0000/dobmk  run_base_test_heredoc.0004/dobmk
run_base_test_heredoc.0001/dobmk  run_base_test_heredoc.0005/dobmk
run_base_test_heredoc.0002/dobmk  run_base_test_heredoc.0006/dobmk
run_base_test_heredoc.0003/dobmk  run_base_test_heredoc.0007/dobmk

==== run_base_test_backslash.0007/dobmk =====
pbind -b 7 $$ >> pbind.out
../run_base_test_backslash.0000/mcf_r_base.backslash inp.in  > inp.out 2>> inp.err

==== run_base_test_fieldN.0007/dobmk =====
pbind -b 7 $$ >> pbind.out
../run_base_test_fieldN.0000/mcf_r_base.fieldN inp.in  > inp.out 2>> inp.err

==== run_base_test_heredoc.0007/dobmk =====
pbind -b 7 $$ >> pbind.out
../run_base_test_heredoc.0000/mcf_r_base.heredoc inp.in  > inp.out 2>> inp.err
$ (Notes about examples) 

IV.C.6. Reporting of submit usage

If you use the submit feature, a notes section will automatically be created to indicate that you have done so.

                                 Submit Notes
                                 ------------
    The config file option 'submit' was used.

You can add notes to that section, or customize it as you wish, by creating lines with notes_submit_NNN. The phrase The config file option 'submit' was used must appear somewhere in your customized notes. You can vary the capitalization of the phrase, you can even break it across multiple lines; it just needs to be present. If it is not, it will automatically be added.

The notes on lines 16-20 appear and disappear automatically depending on whether submit is used.

$ cat -n notes_submit.cfg 
     1  iterations              = 1
     2  output_root             = /tmp/notes_submit
     3  size                    = test
     4  verbose                 = 40
     5  default: # --------- submit stuff ----------------------------------------
     6     bind                 = <<EOT
     7                              2,  3,  5,  7, 11, 13, 17, 19, 23, 29, 31
     8                             37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79
     9                             83, 89, 97
    10  EOT
    11     command_add_redirect = yes
    12     numactlShow          = numactl --show | grep phys >> numactl-s.out 2>&1
    13     submit02             = echo '$[numactlShow]'        > dobmk
    14     submit03             = echo '${command}'           >> dobmk
    15     submit05             = numactl --physcpubind=$BIND sh dobmk
    16     notes_submit_007     =
    17     notes_submit_011     = The config file option
    18     notes_submit_013     = 'submit' was used to prefer
    19     notes_submit_017     = prime processors.
    20     notes_submit_019     =
    21  default: #---------- compiler stuff --------------------------------------
    22     CC                   = gcc
    23     CC_VERSION_OPTION    = -v
    24     OPTIMIZE             = -O
    25  intrate: #---------- suite and benchmark stuff ---------------------------
    26     copies               = 21
    27  intspeed:
    28     use_submit_for_speed = yes
    29  657.xz_s=peak:
    30     EXTRA_OPTIMIZE       = -DSPEC_OPENMP -fopenmp
    31     use_submit_for_speed = no
    32     threads              = 29
    33

The script below uses this config file for three runcpu commands:

1. Run a SPECrate Integer benchmark, 557.xz_r. By default, SPECrate uses submit and therefore config file lines 5-20 apply.
2. Run base for the SPECspeed Integer benchmark 657.xz_s, using submit because of lines 27-28.
3. Run peak for the SPECspeed Integer benchmark 657.xz_s, skipping submit because of line 31.

After the 3 runcpu commands, various results are printed, as explained below.

$ cat notes_submit.sh 
runcpu --config=notes_submit 557.xz_r --tune=base | grep txt
runcpu --config=notes_submit 657.xz_s --tune=base | grep txt
runcpu --config=notes_submit 657.xz_s --tune=peak | grep txt

echo
grep -C1 "to prefer" /tmp/notes_submit/result/CPU2017.00*txt

cd /tmp/notes_submit/benchspec/CPU/557.xz_r/run/
  num_dirs=$(ls -d run* | wc -l)
 num_dobmk=$(ls run*/dobmk | wc -l)
echo
echo 557.xz_r has $num_dirs run dirs and $num_dobmk dobmk scripts

cd /tmp/notes_submit/benchspec/CPU/657.xz_s/run/
      num_dirs_base=$(ls -d run*base* | wc -l)
      num_dirs_peak=$(ls -d run*peak* | wc -l)
 num_dobmk_base=$(ls run*base*/dobmk | wc -l)
 num_dobmk_peak=$(ls run*peak*/dobmk 2>/dev/null | wc -l)
echo
echo "657.xz_s base has $num_dirs_base run dir(s) and $num_dobmk_base dobmk script(s)"
echo "657.xz_s peak has $num_dirs_peak run dir(s) and $num_dobmk_peak dobmk script(s)" 

Notice that when grep searches for "to prefer" in the generated reports, it finds that phrase from notes_submit in CPU2017.001.intrate.test.txt and in CPU2017.002.intspeed.test.txt, but not in CPU2017.003.intspeed.test.txt:

$ ./notes_submit.sh 
    format: Text -> /tmp/notes_submit/result/CPU2017.001.intrate.test.txt
    format: Text -> /tmp/notes_submit/result/CPU2017.002.intspeed.test.txt
    format: Text -> /tmp/notes_submit/result/CPU2017.003.intspeed.test.txt

/tmp/notes_submit/result/CPU2017.001.intrate.test.txt-     The config file option
/tmp/notes_submit/result/CPU2017.001.intrate.test.txt:     'submit' was used to prefer
/tmp/notes_submit/result/CPU2017.001.intrate.test.txt-     prime processors.
--
/tmp/notes_submit/result/CPU2017.002.intspeed.test.txt-     The config file option
/tmp/notes_submit/result/CPU2017.002.intspeed.test.txt:     'submit' was used to prefer
/tmp/notes_submit/result/CPU2017.002.intspeed.test.txt-     prime processors.

557.xz_r has 21 run dirs and 21 dobmk scripts

657.xz_s base has 1 run dir(s) and 1 dobmk script(s)
657.xz_s peak has 1 run dir(s) and 0 dobmk script(s)
$  (Notes about examples) 

The final portion of the output shows that the run directories match the notes.
In short, when submit comes and goes, the notes magically do the same.

V. Config file options for readers

Whether or not you send your result to SPEC, you should fully disclose how you achieved the result. If it requires the installation of the GoFastLinker, you should say so. By setting the appropriate fields in the config file, you can cause information about the GoFastLinker to appear in the reports that are intended for humans.

V.A. Descriptive fields

Here are the fields that you can set to describe your testbed to readers:

Version <id> released MMM-YYYY 

fw_bios = TurboBlasterFW version 20 released Jul-2022

Version <id> of <...> 

fw_management = Version 2.2.3 of Sun Remote System Control (RSC)

hw_line_standard = 120 V / 60 Hz / 1 phase / 2 wire
hw_line_standard = 208 V / 50 Hz / 3 phase / 4 wire

<n> x <id> @ <speed>[...]

hw_nics = 2 x on-board @ 1 GbE; 4 x Mellanox ConnectX 5 @ 200 Gbps

<n> @ <speed> <unit>[...]

hw_nics_connected = 1 @ 1 Gbps

<n> / <m>

hw_nics_enabled = 2 / 1

power_analyzer                    = hex-analyzer001:8888, hex-analyzer002:8888
hw_power_hexanalyzer0018888_label = Base System Power Analyzer
hw_power_hexanalyzer0028888_label = Storage Cabinet Power Analyzer

temp_meter                              = hex-analyzer001:8889, hex-analyzer002:8889
hw_temperature_hexanalyzer0018889_label = Base System Temperature Meter
hw_temperature_hexanalyzer0028889_label = Storage Cabinet Temperature Meter

power_management001 = BIOS and OS set to prefer performance at the cost
power_management002 = of additional power usage.

notes_power_001 = The OS 'poweradm' utility is set to 'disable':
notes_power_003 =    No attempt is made to save power.

notes_power_011 = The BIOS "Energy Mode" option is set to "Performance":
notes_power_012 =    All CPUs are always enabled, all fans are full speed.

      test_sponsor = Genius Compilers
      tester       = Pawtuckaway State College
      hw_vendor    = TurboBlaster 

V.B. Fields can vary by scope.

Fields can appear and disappear based upon scope. For example, if your floating point runs used two Fortran compilers (which is allowed for peak), you could construct a config file that adjusts the fields accordingly:

$ cat tmp.cfg
expand_notes      = 1
hw_vendor         = Turboblaster, Inc.
output_format     = text
output_root       = /tmp/fake
default:
   hw_avail       = Feb-2018
   sw_avail       = Jan-2018
   sw_compiler1   = C: V42.0 of TurboBlaster C/C++
   sw_compiler2   = Fortran: V42.2 of TurboBlaster Fortran
fprate:
   sw_avail       = Apr-2018
   sw_compiler3   = Fortran: V7.3.0 of gfortran
   notes_comp_100 = In base, all benchmarks use Turboblaster Fortran
   notes_comp_110 = In peak, some benchmarks use Turboblaster Fortran and some
   notes_comp_120 =          benchmarks use gfortran, as noted in the report.
$ runcpu --fakereportable --config=tmp intrate fprate | grep rate.txt
    format: Text -> /tmp/fake/result/CPU2017.001.intrate.txt
    format: Text -> /tmp/fake/result/CPU2017.001.fprate.txt
$ cd /tmp/fake/result
$ grep avail CPU2017.001.intrate.txt 
    Test sponsor: Turboblaster, Inc.             Hardware availability: Feb-2018
    Tested by:    Turboblaster, Inc.             Software availability: Jan-2018
$ grep avail CPU2017.001.fprate.txt 
    Test sponsor: Turboblaster, Inc.             Hardware availability: Feb-2018
    Tested by:    Turboblaster, Inc.             Software availability: Apr-2018
$ grep ortran CPU2017.001.intrate.txt 
                      Fortran: V42.2 of TurboBlaster Fortran
$ grep ortran CPU2017.001.fprate.txt 
                      Fortran: V42.2 of TurboBlaster Fortran
                      Fortran: V7.3.0 of gfortran
     In base, all benchmarks use Turboblaster Fortran
     In peak, some benchmarks use Turboblaster Fortran and some
              benchmarks use gfortran, as noted in the report.
$ (Notes about examples)   

In the above example, notice that both the compiler information and the availability date changed in the report, depending on the metric.

V.C. Additional notes for the reader

In addition to the pre-defined fields, you can write as many notes as you wish. These notes are printed in the report, using a fixed-width font. For example, you can use notes to describe software or hardware information with more detail beyond the predefined fields:

   notes_os_001  = The operating system used service pack 2 plus patches
   notes_os_002  = 31415, 92653, and 58979.  At installation time, the
   notes_os_003  = optional "Numa Performance Package" was selected.

V.C.1. Notes sections

There are various notes sections. If there are no notes in a particular section, it is not output, so you don't need to worry about making sure you have something in each section.

The sections, in order of appearance, are as follows:

1. notes_comp_NNN -- Notes about compiler invocation.
2. notes_port_NNN -- Notes about portability options.
3. notes_base_NNN -- Notes about base optimization options.
4. notes_peak_NNN -- Notes about peak optimization options.
5. notes_submit_NNN -- Notes about use of the submit option.
6. notes_os_NNN -- Notes about operating system tuning and changes.
7. notes_part_NNN -- Notes about component parts (for kit-built systems).
8. notes_NNN -- General notes.
9. notes_plat_NNN -- Notes about platform tuning and changes.
10. notes_power_NNN -- Notes about power-related hardware and settings.

Notes about the submit command are described above, with the description of the submit option.

V.C.2. Note numbering

Start your notes with the name of the notes section where you want the note to appear, and then add numbers to define the order of the lines. Within a section, notes are sorted by line number. The NNN above is not intended to indicate that you are restricted to 3 digits; you can use a smaller or larger number of digits as you wish, and you can skip around as you like: for example, ex-BASIC programmers might naturally use line numbers 100, 110, 120... But note that if you say notes_plat782348320742972403 you just might encounter the dreaded (and highly unusual) "out of memory" error, so don't do that.

You can optionally include an underscore just before the number, but beware: if you say both notes_plat_105 and notes_plat105, both are considered to be the same line. The last one mentioned will replace the first, and it will be the only one output.

V.C.3. Additional tags

For all sections you can add an optional additional tag of your choosing before the numbers. Notes will be organized within the tags.

The intent of the feature is that it may allow you to organize your system information in a manner that better suits your own categories for describing it.

For example:

$ cat notes_tags.cfg 
iterations      = 1
output_format   = text
output_root     = /tmp/notes_tags
runlist         = 519.lbm_r
size            = test
fprate=base:
   CC                       = gcc
   CC_VERSION_OPTION        = -v
   notes_part_greeting_011  = ++ how
   notes_part_greeting_20   = ++ you?
   notes_part_greeting_012  = ++ are
   notes_part_aname_1       = ++ Alex,
   notes_part_080           = ++ hi

$ cat notes_tags.sh  
runcpu --config=notes_tags | grep txt
grep '++' /tmp/notes_tags/result/CPU2017.001*txt
$ ./notes_tags.sh
    format: Text -> /tmp/notes_tags/result/CPU2017.001.fprate.test.txt
     ++ hi
     ++ Alex,
     ++ how
     ++ are
     ++ you?
$ (Notes about examples) 

V.C.4. Links in notes sections

You can mention URLs in your notes section, and html reports will correctly render them as hyperlinks. For example:

notes_plat_001 = Additional detail may be found at
notes_plat_002 = http://www.turboblaster.com/servers/big/green/

If you like, you can use descriptive text for the link by preceding it by the word LINK and adding the descriptive text in square brackets:

LINK url AS [descriptive text]

The brackets may be omitted if your descriptive text is a single word, without blanks.

For example:

notes_plat_001 = Additional detail may be found at
notes_plat_002 = LINK http://www.turboblaster.com/servers/big/green/ AS [TurboBlaster Servers]

When the above appears in an html report, it is rendered as:

 Additional detail may be found at
 TurboBlaster Servers
      

And in a text report, it appears as:

                                Platform Notes
                                --------------
     Additional detail may found at
     TurboBlaster Servers (http://www.turboblaster.com/servers/big/green/)

Since the text report is not a context in which the reader can click on a link, it is spelled out instead. Note that because the text report spells the link out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to pick whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall of the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the link is spelled out.

V.D. About Parallel Reporting

If benchmarks are optimized to use multiple threads, cores, and/or chips at run time, this is reported via the phrase:

Parallel: Yes

The value for "Parallel" is derived from the parallel flag attribute for compiler flags that cause binaries to be multi-threaded.

VI. Using Feedback Directed Optimization (FDO)

Feedback Directed Optimization (FDO) is an optimization method that uses multiple steps, typically:

• Build a program.
• Run a sample "training" workload, collecting a profile as it runs.
• Apply the profile to produce a modified program.

FDO is also sometimes known as PBO, for Profile-Based Optimization.

This section explores FDO controls and provides examples.
The controls are:

1. PASSnmakevars (summary)  (full list)
2. fdo shell options.
3. feedback config file option.
4. --feedback runcpu option.

All examples in this section use --fake, which is especially recommended when debugging FDO commands. You can send fake's very wordy output to a file to study it, or subset it with commands such as grep or findstr.

VI.A. Minimum required: PASSn or fdo_

To use feedback, you must use either a PASSn make variable, which adds flags to a pre-defined sequence of FDO build steps; or an fdo_ shell option, which lets you modify and add to the sequence of FDO steps.

PASSn: The most common way of using Feedback Directed Optimization is by setting PASSn makevars (summary)  (full list).
The sequence is:

• build  -   Create an instrumented binary using the config file PASS1 flags.
• train  -   Run it with the SPEC-provided training workload. The instrumentation collects a profile.
• rebuild  -   Create a new binary using the config file PASS2 flags. The compiler uses the profile to improve its optimizations.

$ cat fdoExample1.cfg 
action          = build
runlist         = 549.fotonik3d_r
tune            = peak
default=peak:
   FC           = tbf90
   PASS1_FFLAGS = --CollectFeedback
   PASS2_FFLAGS = --ApplyFeedback

This example config file use Turboblaster Fortran 90 to build 549.fotonik3d_r specifying PASS1 and PASS2 flags.
A script picks out a few lines from the log file, including Fortran compile commands for source module readline.f90.

$ cat fdoExample1.sh 
runcpu --config=fdoExample1 --fake | grep -e readline.f90 -e Train
$ ./fdoExample1.sh 
tbf90 -c -o readline.o -I. --CollectFeedback readline.f90
Training 549.fotonik3d_r with the train workload
tbf90 -c -o readline.o -I. --ApplyFeedback readline.f90
$  (Notes about examples) 

fdo_: Feedback Directed Optimization can also be done by setting up up fdo shell commands.
For example:

• build  -   Build just once.
• train  -   Run it with the SPEC-provided training workload.
• do something else   -   Perhaps you have a post-link optimizer that applies a profile without needing to recompile.

$ cat fdoExample2.cfg 
action          = build
runlist         = 549.fotonik3d_r
tune            = peak
default=peak:
   FC           = tbf90
   OPTIMIZE  = -fast -profile:fbdir
   fdo_post1 = /opt/bin/postoptimizer --profile:fbdir

Notice that this config file does not mention PASS1, but does use fdo_post1.
The script below picks out a few lines of interest from the log:

$ cat fdoExample2.sh 
runcpu --config=fdoExample2 --fake \
   | grep -e readline.f90 -e Train -e '^/opt'
$ ./fdoExample2.sh 
tbf90 -c -o readline.o -I. -fast -profile:fbdir readline.f90
Training 549.fotonik3d_r with the train workload
/opt/bin/postoptimizer --profile:fbdir
$ (Notes about examples) 

Unlike the previous example, the grep command finds only one compile for readline.f90.

VI.B. Flexible build models

You can adjust the fdo commands that are generated. Perhaps the default model is almost correct for your needs, and you just want minor changes. Caution: Before changing an fdo_ option, find its current setting using --fake. Check it in the context of your config file: the commands will vary.

Goal: discover exactly what the SPEC tools are doing during pass 2 cleanup.

$ cat fdoExample3.cfg 
action          = build
output_root     = /tmp/fake
runlist         = 549.fotonik3d_r
tune            = peak
default=peak:
   FC           = tbf90
   PASS1_FFLAGS = --CollectFeedback
   PASS2_FFLAGS = --ApplyFeedback

The config file sends output to an output_root.
The script below searches the log file in the result/ directory under that root. The command grep -n prints numbered lines matching '%% Fake commands' and the cut command prints out the first 5 words from matching lines.

$ cat fdoExample3.sh 
runcpu --config=fdoExample3 --fake | grep "log for"
grep -n '%% Fake commands' /tmp/fake/result/CPU2017.001.log \
    | cut -f 1-5 -d' '
$
$ ./fdoExample3.sh 
The log for this run is in /tmp/fake/result/CPU2017.001.log
387:%% Fake commands from make.clean
397:%% Fake commands from fdo_make_pass1
438:%% Fake commands from options1
482:%% Fake commands from compiler-version1
496:%% Fake commands from input_generation
611:%% Fake commands from benchmark_run
720:%% Fake commands from compare_run
819:%% Fake commands from fdo_make_clean_pass2
828:%% Fake commands from fdo_make_pass2
869:%% Fake commands from options2
913:%% Fake commands from compiler-version2

Suppose we are specifically interested in what happens during fdo_make_clean_pass2, line 819 in the above list.
To see the details, look just before and just after the matching phrase:

$ cat -n /tmp/fake/result/CPU2017.001.log | head -825 | tail -10 
   816
   817  specmake -n --output-sync fdoclean FDO=PASS2
   818
   819  %% Fake commands from fdo_make_clean_pass2 (specmake -n --output-sync fdoclean FDO=P...):
   820  rm -rf *.o  pscyee.out
   821  find . \( -name \*.o -o -name '*.fppized.f*' -o -name '*.i' -o -name '*.mod' \) -print | xargs rm -rf
   822  rm -rf fotonik3d_r
   823  rm -rf fotonik3d_r.exe
   824  %% End of fake output from fdo_make_clean_pass2 (specmake -n --output-sync fdoclean FDO=P...)
   825
$ (Notes about examples) 

To decipher the above:

• Lines 820 to 823 show that various files are removed.
• Line 819 tells us that they were removed because fdo_make_clean_pass2 did it.
• Line 819 also tells us that fdo_make_clean_pass2 begins with the phrase in parentheses (specmake -n --output-sync fdoclean FDO=P...).
• To see the full command, look 2 lines up, at line 817.

Combining PASSn and fdo_ You can combine PASSn and fdo_ while adjusting the feedback commands.
For example, suppose that your compiler does not need to recompile all the modules in step 2; you just want to relink.
Difficulty: by default, build #2 deletes all the object files.

$ cat -n fdoExample4.cfg 
     1  action                   = build
     2  runlist                  = 549.fotonik3d_r
     3  tune                     = peak
     4  default=peak:
     5     FC                   = pgf90
     6     PASS1_LDFLAGS        = -PGINSTRUMENT -incremental:no
     7     PASS2_LDFLAGS        = -PGOPTIMIZE   -incremental:no
     8     fdo_make_clean_pass2 = rm ${baseexe}
     9     fdo_make_pass2       = specmake build FDO=PASS2

On line 8, the default cleaning action (from Example 3) is changed to remove only the actual executable. Under the usual rules of GNU make, you might expect that would be sufficient to cause build #2 to do only the link step. In this case, it is not, because by default the tools generate specmake --always-make. Line 9 above overrides that default.

$ cat fdoExample4.sh 
runcpu --config=fdoExample4 --fake \
   | grep -e readline.f90          \
          -e Train                 \
          -e '^rm'                 \
          -e '^specmake.*clean'    \
          -e '^specmake.*build'    \
   | cat -n

The runcpu command above is sent to grep to pick lines of interest, which are numbered by cat.
In the output, lines 1-5 show a typical clean action; much less is removed on line 10. Compare line 6 to line 11 to see the differences in specmake build options.

$ ./fdoExample4.sh 
     1  specmake -n --output-sync clean
     2  rm -rf *.o  pscyee.out
     3  rm -rf fotonik3d_r
     4  rm -rf fotonik3d_r.exe
     5  rm -rf core
     6  specmake -n --output-sync --always-make build FDO=PASS1
     7  pgf90 -c -o readline.o -I. readline.f90
     8  rm -rf options.tmpout
     9  Training 549.fotonik3d_r with the train workload
    10  rm fotonik3d_r
    11  specmake build FDO=PASS2
$  (Notes about examples) 

Adding FDO steps: You can add more steps.

For example, you could:

• Create runcpu variables with customized content (lines 6-9)
• Build twice with different types of collectors (PASSn_OPTIMIZE on lines 13 and 17 below)
• Train twice (lines 14 and 18)
• Combine profiles in temporary files (lines 9, 15, 19)
• Build a third time using the combined profile (line 22)
• Instrument with a postoptimizer, train a third time, and optimize again (lines 23-25)

$ cat -n fdoExample5.cfg 
     1  action                = build
     2  command_add_redirect  = yes
     3  runlist               = 641.leela_s
     4  tune                  = peak
     5  default=peak:
     6      profdir           = /tmp/feedback/profiles
     7      big_profile       = ${profdir}/${benchnum}.aggregated.profile
     8      clean_profile     = mkdir -p ${profdir}; rm -f ${big_profile}
     9      append_to_profile = cat this.prof >> ${big_profile}
    10      #
    11      fdo_pre0          = ${clean_profile}
    12      #
    13      PASS1_OPTIMIZE    = --collect:paths
    14      fdo_run1          = ${command}               # profile program paths
    15      fdo_post1         = ${append_to_profile}
    16      #
    17      PASS2_OPTIMIZE    = --collect:dcache
    18      fdo_run2          = ${command}               # profile data patterns
    19      fdo_post2         = ${append_to_profile}
    20      #
    21      fdo_pre_make3     = mv ${big_profile} ./profile.in
    22      PASS3_OPTIMIZE    = --apply:paths,dcache
    23      fdo_post_make3    = postopt --instrument:icache
    24      fdo_run3          = ${command}               # profile icache packing
    25      fdo_post3         = postopt --fixup:icache

The script below picks out lines of interest using grep.

$ cat fdoExample5.sh 
runcpu --fake --config=fdoExample5 \
  | grep -e '^mkdir'               \
         -e FullBoard.cpp          \
         -e Train                  \
         -e this.prof              \
         -e ^mv                    \
         -e '^postopt'             \
  | cat -n

The three instances of PASSn_OPTIMIZE are carried out on lines 2, 5, and 9 below.
Various manipulations of the profile files are shown on lines 1, 4, 7, and 8.

$ ./fdoExample5.sh 
     1  mkdir -p /tmp/feedback/profiles; rm -f /tmp/feedback/profiles/641.aggregated.profile
     2  CC -c -o FullBoard.o -DSPEC -DSPEC_CPU -DNDEBUG -I.      --collect:paths         FullBoard.cpp
     3  Training 641.leela_s with the train workload
     4  cat this.prof >> /tmp/feedback/profiles/641.aggregated.profile
     5  CC -c -o FullBoard.o -DSPEC -DSPEC_CPU -DNDEBUG -I.      --collect:dcache         FullBoard.cpp
     6  Training 641.leela_s with the train workload
     7  cat this.prof >> /tmp/feedback/profiles/641.aggregated.profile
     8  mv /tmp/feedback/profiles/641.aggregated.profile ./profile.in
     9  CC -c -o FullBoard.o -DSPEC -DSPEC_CPU -DNDEBUG -I.      --apply:paths,dcache         FullBoard.cpp
    10  postopt --instrument:icache
    11  Training 641.leela_s with the train workload
    12  postopt --fixup:icache
$ (Notes about examples) 

• Tip 1: If you try the above example, send the fake output to a file, and examine it with an editor.
• Tip 2: You might be tempted to create the profile file in fdo_run1 (config file line 13) using a single angle bracket, and append to it in fdo_run2 (config line 17) with double angle brackets. Don't do that. Some benchmarks are invoked multiple times when they are trained, therefore all instances of fdo_run should append. You can clear the file before the build using fdo_pre0 (config line 10).

VI.C. The config file feedback option

If you use PASSn or fdo_ then, by default, FDO is used for peak builds. The config file option feedback provides an additional control, an "on/off" switch that can be applied by the usual rules of precedence.

A common usage model is to enable feedback everywhere, then turn it off selectively:

$ cat -n fdoExample6.cfg 
  1  action         = build
  2  label          = miriam
  3  runlist        = 503.bwaves_r,519.lbm_r,549.fotonik3d_r
  4  tune           = base,peak
  5  fprate:
  6     fdo_post1   = /opt/merge_feedback
  7  519.lbm_r:
  8     feedback    = no
$ cat fdoExample6.sh 
runcpu --fake --config=fdoExample6 | grep -e Building -e ^/opt
$ ./fdoExample6.sh 
  Building 503.bwaves_r base miriam: (build_base_miriam.0000) [2017-02-23 22:31:30]
  Building 519.lbm_r base miriam: (build_base_miriam.0000) [2017-02-23 22:31:31]
  Building 549.fotonik3d_r base miriam: (build_base_miriam.0000) [2017-02-23 22:31:32]
  Building 503.bwaves_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:31:33]
/opt/merge_feedback
  Building 519.lbm_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:31:35]
  Building 549.fotonik3d_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:31:35]
/opt/merge_feedback
$ (Notes about examples) 

Lines 5-6 turn feedback on for SPECrate floating point benchmarks. Neverthelss, 519.lbm_r peak does not use FDO: line 8 takes priority over line 6 by the usual rules of precedence. Feedback is not used for base, because the tools are aware of the rule that disallows it.

VI.D. runcpu --feedback

If you use the runcpu option --feedback or its opposite, --nofeedback, all peak benchmarks are affected.
New with CPU 2017: The command line wins unconditionally over the config file.

The example below uses the same config file as the previous example.

$ cat fdoExample7.sh 
echo build with --feedback
runcpu --fake --config=fdoExample6 --feedback | grep -e Building -e '^/opt'
echo
echo build with --nofeedback
runcpu --fake --config=fdoExample6 --nofeedback | grep -e Building -e '^/opt'
$ ./fdoExample7.sh 
build with --feedback
  Building 503.bwaves_r base miriam: (build_base_miriam.0000) [2017-02-23 22:37:07]
  Building 519.lbm_r base miriam: (build_base_miriam.0000) [2017-02-23 22:37:08]
  Building 549.fotonik3d_r base miriam: (build_base_miriam.0000) [2017-02-23 22:37:09]
  Building 503.bwaves_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:37:10]
/opt/merge_feedback
  Building 519.lbm_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:37:11]
/opt/merge_feedback
  Building 549.fotonik3d_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:37:12]
/opt/merge_feedback

build with --nofeedback
  Building 503.bwaves_r base miriam: (build_base_miriam.0000) [2017-02-23 22:37:16]
  Building 519.lbm_r base miriam: (build_base_miriam.0000) [2017-02-23 22:37:17]
  Building 549.fotonik3d_r base miriam: (build_base_miriam.0000) [2017-02-23 22:37:18]
  Building 503.bwaves_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:37:19]
  Building 519.lbm_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:37:20]
  Building 549.fotonik3d_r peak miriam: (build_peak_miriam.0000) [2017-02-23 22:37:20]
$ (Notes about examples) 

In the run with --feedback, all peak benchmarks use feedback. Base does not, because the tools are aware of the rule.
The run with --nofeedback does not use FDO for any benchmarks.

VII. The config file preprocessor

VII.A. Introduction

$ cat -n preprocessor_example1.cfg
 1  500.perlbench_r,600.perlbench_s:
 2  %if %{chip} eq "sparc"
 3      PORTABILITY    = -DSPEC_SOLARIS_SPARC
 4  %elif %{chip} eq "x86"
 5  %   if %{bits} == 32
 6  %      define suffix IA32
 7  %   else
 8  %      define suffix X64
 9  %   endif
10      PORTABILITY    = -DSPEC_SOLARIS_%{suffix}
11  %endif
$ (Notes about examples) 

$ configpp -c preprocessor_example1.cfg --define chip=sparc | grep -e perlbench -e PORT
500.perlbench_r,600.perlbench_s:
    PORTABILITY    = -DSPEC_SOLARIS_SPARC

$ configpp -c preprocessor_example1.cfg --define chip=x86 | grep -e perlbench -e PORT
500.perlbench_r,600.perlbench_s:
    PORTABILITY    = -DSPEC_SOLARIS_X64

$ configpp -c preprocessor_example1.cfg -S chip=x86 -S bits=32 | grep -e perlbench -e PORT
500.perlbench_r,600.perlbench_s:
    PORTABILITY    = -DSPEC_SOLARIS_IA32
            

VII.B. Preprocessor Syntax Basics

VII.B.1. Column 1. Always Punch Column 1

%define foo
%    define bar
%		undef hello 

 %define foo           Space in the first column
        %define foo    Tab in the first column
#define foo            Use % in column 1, not #  

VII.B.2. One line per directive

VII.B.3. Comments

The usual rules for comments DO apply: anything after # will be ignored.

VII.C. Macro Definition

The next 3 sections describe how to define, undefine, or redefine macros.

VII.C.1. Defining Macros

%define macro_name value

$ runcpu --define mymacro=something
$ runcpu -S mymacro=something 

VII.C.2. Undefining macros

VII.C.3. Redefining macros

%if !defined %{mypath}
%   define mypath "/usr/bin/"
%endif

%if defined %{mypath}
%   undef mypath
%   define mypath "/usr/bin/"
%endif

VII.D. Macro Usage

 %{foo} 

$ cat noSuchMacro.cfg 
%define Type -fast
CXXOPTIMIZE = %{Type}
COPTIMIZE   = %{Typo}
$ configpp -c noSuchMacro.cfg | grep OPTIMIZE
CXXOPTIMIZE = -fast
COPTIMIZE   = %{Typo}
$ (Notes about examples) 

$ cat macroDefinedButNoValue.cfg 
%define baz
%ifdef %{baz}
%   info baz is defined
%   info In text context it is: "%{baz}".
%endif
%if %{baz}
%   info In logical context, it behaves as true.
%endif
%if %{baz} + 3  == 4
%   info In numeric context, it behaves as 1.
%endif
$ configpp --config=macroDefinedButNoValue | grep INFO
  INFO: baz is defined
  INFO: In text context it is: "".
  INFO: In logical context, it behaves as true.
  INFO: In numeric context, it behaves as 1.
$  (Notes about examples) 

$ cat macroEmptyValue.cfg 
%define baz ""
%ifdef %{baz}
%   info baz is defined
%   info In text context it is: "%{baz}".
%endif
%if ! %{baz}
%   info In logical context, it behaves as false
%endif
%if %{baz} + 3  == 3
%   info In numeric context, it behaves as 0
%endif
$ configpp --config=macroEmptyValue | grep INFO
  INFO: baz is defined
  INFO: In text context it is: "".
  INFO: In logical context, it behaves as false
  INFO: In numeric context, it behaves as 0
$ (Notes about examples) 

$ cat silly_macros.cfg 
%define foo       Hello_
%define bar       baz
%define foobar    Huh?
%define foobaz    What?
%define Hello_baz Please don't do this

OPTIMIZE  = %{foo}
COPTIMIZE = %{foo%{bar}}
FOPTIMIZE = %{%{foo}%{bar}}
$ configpp -c silly_macros.cfg | grep OPTIMIZE
OPTIMIZE  = Hello_
COPTIMIZE = What?
FOPTIMIZE = Please don't do this
$ (Notes about examples) 

VII.E. Conditionals

Conditionals provide the ability to include and exclude entire sections of text based on macros and their values.

VII.E.1. %ifdef

$ cat xyzzy.cfg 
%define foo
%ifdef %{foo}
   OPTIMIZE = This text will be included
%endif
%ifdef %{bar}
   FOPTIMIZE = This text will not be included
%endif
$ configpp --config=xyzzy | grep include
   OPTIMIZE = This text will be included
$

VII.E.2. %ifndef

VII.E.3. %if and expression evaluation

%if expression
...
%endif

%if defined(%{foo}) && !defined(%{bar}) || %{baz} == 3

%if defined(%{foo}) and !defined(%{bar}) or %{baz} == 3

%if %{foo} eq 'Hello, Dave.'

%if %{foo} * 2004 > 3737

%if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
%   error Please use --define chip=sparc or --define chip=x86
%endif

$ cat test_define.cfg
%ifdef %{a}
   OPTIMIZE = -a
%endif
%if defined(%{b})
   LDOPTIMIZE = -b
%endif
%if %{c}
   COPTIMIZE = -c
%endif
$ configpp --config=test_define -S a=1 -S b=1 -S c=1 | grep OPT 
   OPTIMIZE = -a
   LDOPTIMIZE = -b
   COPTIMIZE = -c
$ configpp --config=test_define -S a=0 -S b=0 -S c=0 | grep OPT
   OPTIMIZE = -a
   LDOPTIMIZE = -b

$ cat pathnote.cfg
%define pathnote substr("%{ENV_PATH}",0,index("%{ENV_PATH}",":"))
%info my path begins %{pathnote}
$ export PATH=/usr/local/something/something:$PATH
$ configpp --config=pathnote | grep INFO
  INFO: my path begins /usr/local/something/something
$  (Notes about examples) 

VII.E.4. %else

$ cat tmp.cfg
%ifdef %{foo}
   OPTIMIZE = --foo
%else
   OPTIMIZE = --nofoo
%endif
$ configpp --config=tmp | grep OPTIM
   OPTIMIZE = --nofoo
$

VII.E.5. %elif

$ cat tmp.cfg
%define foo Hello!
%if !defined(%{foo})
%   info This text will not be included
%elif defined(%{bar})
%   info This text won't be included either
%elif '%{foo}' eq 'Hello!'
%   info This text WILL be included
%else
%   info Alas, the else here is left out as well.
%endif
$ configpp --config=tmp | grep INFO 
  INFO: This text WILL be included
$  (Notes about examples) 

VII.E.6. Preprocessor Example: Picking CPUs

$ cat hidekatsu.cfg 
include: TurboBlaster9000.inc

503.bwaves_r=peak:
   copies      = %{Use1PerCore}
557.xz_r=peak:
   copies      = %{Use2PerCore}

603.bwaves_s=peak:
   threads     = %{Use3PerCore}
657.xz_s=peak:
   threads     = %{Use2PerCore}
$ 

$ cat -n TurboBlaster9000.inc 
  1  # TurboBlaster 9000 has:
  2  #   1 to 4    chips per system
  3  #   2         cores per chip
  4  #   3         hardware threads (virtual CPUs) per core
  5
  6  %ifndef %{chips}
  7  %   error Please say runcpu --define chip=N
  8  %endif
  9
 10  %define   cores        2 * %{chips}
 11
 12  %define   Use1PerCore  1 * %{cores}
 13  %define   Use2PerCore  2 * %{cores}
 14  %define   Use3PerCore  3 * %{cores}
 15
 16  # Use 'bind' to place jobs on alternating chips
 17  # and alternating cores.
 18
 19         # core 0           core 1
 20  %if %{chips} == 1
 21     bind=\
 22           0                3          \
 23           1                4          \
 24           2                5
 25  %elif %{chips} == 2
 26     bind=\
 27           0  6             3  9       \
 28           1  7             4 10       \
 29           2  8             5 11
 30  %elif %{chips} == 3
 31     bind=\
 32           0  6 12          3  9 15    \
 33           1  7 13          4 10 16    \
 34           2  8 14          5 11 17
 35  %elif %{chips} == 4
 36     bind=\
 37           0  6 12 18       3  9 15 21 \
 38           1  7 13 19       4 10 16 22 \
 39           2  8 14 20       5 11 17 23
 40  %endif 

$ cat hidekatsu.cfg 
include: TurboBlaster9000.inc

503.bwaves_r=peak:
   copies      = %{Use1PerCore}
557.xz_r=peak:
   copies      = %{Use2PerCore}

603.bwaves_s=peak:
   threads     = %{Use3PerCore}
657.xz_s=peak:
   threads     = %{Use2PerCore}
$
            

$ configpp -c hidekatsu \
  --output=one.out \
  --define chips=1

  ...

$ tail -17 one.out.cfg 
   bind=\
    0      3   \
    1      4   \
    2      5

503.bwaves_r=peak:
   copies   = 2
557.xz_r=peak:
   copies   = 4

603.bwaves_s=peak:
   threads  = 6
657.xz_s=peak:
   threads  = 4
            

$ configpp -c hidekatsu \
  --output=two.out \
  --define chips=2

   ...

$ tail -17 two.out.cfg 
   bind=\
    0  6    3  9  \
    1  7    4 10  \
    2  8    5 11

503.bwaves_r=peak:
   copies   = 4
557.xz_r=peak:
   copies   = 8

603.bwaves_s=peak:
   threads  = 12
657.xz_s=peak:
   threads  = 8
            

$ configpp -c hidekatsu \
  --output=four.out \
  --define chips=4

  ...

$ tail -17 four.out.cfg 
   bind=\
   0  6 12 18   3  9 15 21 \
   1  7 13 19   4 10 16 22 \
   2  8 14 20   5 11 17 23

503.bwaves_r=peak:
   copies   = 8
557.xz_r=peak:
   copies   = 16

603.bwaves_s=peak:
   threads  = 24
657.xz_s=peak:
   threads  = 16
            

VII.F. Informational Directives

It's often helpful to be able to warn or exit on certain conditions. Perhaps there's a macro that must be set to a particular value, or maybe it's just very highly recommended.

VII.F.1. %warning

$ cat warning.cfg 
%if !defined(%{somewhat_important_macro})
%   warning You have not defined somewhat_important_macro!
%endif
$
$ cat warning.sh 
configpp --config=warning | grep -C2 WARNING
$
$ ./warning.sh 

***
  WARNING: You have not defined somewhat_important_macro!
           (From line 2 of
           /cpu2017/rc4/config/warning.cfg.)
$ (Notes about examples) 

VII.F.2. %error

$ cat error.cfg 
%if !defined(%{REALLY_important_macro})
%   error You have not defined REALLY_important_macro!
%endif
$
$ cat error.sh 
configpp --config=error > /tmp/out
echo runcpu exit code: $?
grep -C2 ERROR /tmp/out
$ ./error.sh 
runcpu exit code: 1

************************************************************************
  ERROR: You have not defined REALLY_important_macro!
         (From line 2 of
         /Users/jhenning/spec/cpu2017/rc4/config/error.cfg.)
$ (Notes about examples) 

VII.F.3. %info

$ cat -n info.cfg 
   1  %if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
   2  %   error Please use --define chip=sparc or --define chip=x86
   3  %endif
   4
   5  %if  %{chip} eq "sparc"
   6  %   define default_build_ncpus 64
   7  %elif %{chip} eq "x86"
   8  %   define default_build_ncpus 20
   9  %endif
  10  %ifndef   %{build_ncpus}
  11  %   define  build_ncpus   %{default_build_ncpus}
  12  %endif
  13
  14  %info Preprocessor selections:
  15  %info    .    build_ncpus      %{build_ncpus}
  16  %info    .    chip             %{chip}
  17
  18  makeflags = --jobs=%{build_ncpus}
$ configpp -c info --define chip=sparc | grep -e make -e INFO
  INFO: Preprocessor selections:
  INFO: .    build_ncpus      64
  INFO: .    chip             sparc
makeflags = --jobs=64
$ (Notes about examples) 

VII.F.4. %dumpmacros

$ cat -n dumpmacros.cfg 
  1  %if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
  2  %   error Please use --define chip=sparc or --define chip=x86
  3  %endif
  4
  5  %if  %{chip} eq "sparc"
  6  %   define default_build_ncpus 64
  7  %elif %{chip} eq "x86"
  8  %   define default_build_ncpus 20
  9  %endif
 10  %ifndef   %{build_ncpus}
 11  %   define  build_ncpus   %{default_build_ncpus}
 12  %endif
 13
 14  %dumpmacros
$
$ configpp -c dumpmacros --define chip=sparc | grep -e DBG
  DBG: build_ncpus: '%{default_build_ncpus}'
  DBG: chip: 'sparc'
  DBG: default_build_ncpus: '64'
  DBG: runcpu: 'configpp -c dumpmacros --define chip=sparc'
  .
  .
  .
$ 

VII.G. Predefined macros and the environment

$ cat printPath.cfg 
%info %{ENV_PATH}
$ configpp --config=printPath | grep INFO:
  INFO: /spec/cpu2017/rc4/bin:/usr/local/bin:/bin/:/usr/
$  (Notes about examples) 

VII.G.1. Automating output_root

You can automatically bring an output_root into your config file using %{ENV_GO} and navigate around it, as shown in the example for the ogo utility.

VII.G.2. Example: Adjusting the Environment

$ cat -n adjustEnv.cfg
     1  action       = build
     2  output_root  = /tmp
     3  rebuild      = 1
     4  runlist      = 519.lbm_r
     5  verbose      = 99
     6
     7  %define fivetop  %{ENV_HOME}/work/compilers/gcc-5.3.0
     8  %define eighttop %{ENV_HOME}/work/compilers/gcc-8.1.0
     9
    10  %ifdef %{wantGccV8}
    11     preENV_PATH            = %{eighttop}/bin:%{ENV_PATH}
    12     preENV_LD_LIBRARY_PATH = %{eighttop}/lib64:%{ENV_LD_LIBRARY_PATH}
    13  % else
    14     preENV_PATH            = %{fivetop}/bin:%{ENV_PATH}
    15     preENV_LD_LIBRARY_PATH = %{fivetop}/lib64:%{ENV_LD_LIBRARY_PATH}
    16  %endif
    17
    18  default:
    19  CC                = gcc
    20  CC_VERSION_OPTION = -v
   

$ cat adjustEnv.sh 
export LD_LIBRARY_PATH=my:old:path
runcpu  -c adjustEnv                     | grep -e 'Setting LD_LIBRARY_PATH' -e 'gcc version'
runcpu  -c adjustEnv --define wantGccV8  | grep -e 'Setting LD_LIBRARY_PATH' -e 'gcc version'
$ ./adjustEnv.sh 
Setting LD_LIBRARY_PATH = "/Users/jhenning/work/compilers/gcc-5.3.0/lib64:"
gcc version 5.3.0 (JLH  04-Dec-2016)
gcc version 5.3.0 (JLH  04-Dec-2016)
Setting LD_LIBRARY_PATH = "/Users/jhenning/work/compilers/gcc-8.1.0/lib64:"
gcc version 8.1.0 (jhenning-02-May-2018)
gcc version 8.1.0 (jhenning-02-May-2018)
$  (Notes about examples) 

VIII. Output files - and how they relate to your config file

This section describes how the location and contents of several kinds of output files are influenced by your config file.

VIII.A. Help, I've got too many config files!

It was mentioned above that the HASH section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

$ cd $SPEC/config
$ ls tune.cfg*
tune.cfg                        tune.cfg.2017-02-05T124831      tune.cfg.2017-02-05T125733
tune.cfg.2017-02-05T120242      tune.cfg.2017-02-05T125557      tune.cfg.2017-02-05T125738
tune.cfg.2017-02-05T122021      tune.cfg.2017-02-05T125603      tune.cfg.2017-02-05T125744
tune.cfg.2017-02-05T122026      tune.cfg.2017-02-05T125608      tune.cfg.2017-02-05T125749
tune.cfg.2017-02-05T124215      tune.cfg.2017-02-05T125614      tune.cfg.2017-02-05T125756
tune.cfg.2017-02-05T124222      tune.cfg.2017-02-05T125620      tune.cfg.2017-02-05T125802
tune.cfg.2017-02-05T124728      tune.cfg.2017-02-05T125626      tune.cfg.2017-02-05T125807
tune.cfg.2017-02-05T124739      tune.cfg.2017-02-05T125632      tune.cfg.2017-02-05T125812
tune.cfg.2017-02-05T124821      tune.cfg.2017-02-05T125727
$  (Notes about examples) 

If this feels like too much clutter, you can disable the backup mechanism, as described under backup_config. Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove just portions of the clutter, with selective removal of older version; or sweep them to a wastebasket every now and then:

$ cd $SPEC/config
$ mkdir wastebasket
$ mv *cfg.2017* wastebasket/
$

VIII.B. The log file and verbosity levels

$SPEC/result (Unix) or %SPEC%\result (Windows) contains reports and log files. When you are doing a build, you will probably find that you want to pay close attention to the log files such as CPU2017.001.log. Depending on the verbosity level that you have selected, it will contain detailed information about how your build went.

The CPU 2017 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information that are probably useful only to tools developers (level 99). Selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

VIII.B.1. Useful Search Strings

When you are trying to find your way through a log file, you will probably find these (case-sensitive) search strings useful:

VIII.B.2. About Temporary Debug Logs

There are also temporary debug logs, such as CPU2017.001.log.debug. A debug log contains very detailed debugging output from the SPEC tools, as if --verbose 99 had been specified.

For a successful run, the debug log will be removed automatically, unless you specify "--keeptmp" on the command line, or "keeptmp=yes" in your config file.

For a failed run, the debug log is kept. The debug log may seem overwhelmingly wordy, repetitive, detailed, redundant, repetitive, and long-winded, and therefore useless. Suggestion: after a failure, try looking in the regular log first, which has a default verbosity level of 5. If your regular log doesn't have as much detail as you wish, then you can examine the additional detail in the debug log.

If you file a support request, you may be asked to send in the debug log.

VIII.B.3. Verbosity levels

The 'level' referred to in the table below is selected either in the config file verbose option or in the runcpu command as in 'runcpu --verbose n'.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 5. This means that on your screen you will get messages at levels 0 through 5, and 100 through 105. In your log file, you'll find the same messages, plus the messages at levels 106 through 199.

VIII.C. Deciphering an FDO log file

This section demonstrates how to find various portions of a log file using an example build with feedback-directed optimization (FDO). See the section Using Feedback for more information on how to enable FDO.

FDO typically requires two compiles. The first creates an executable image with instrumentation. You run the program with a "training" workload, the instrumentation observes it, and a profile is written. The second compile then uses the profile to improve optimization. SPEC CPU makes all of this relatively easy. Here's a config file that builds 519.lbm_r with FDO:

$ cat mat.cfg
iterations            = 1
label                 = blue271
runlist               = 519.lbm_r
size                  = test
teeout                = yes
tune                  = peak
default:
   CC                 = gcc
   CC_VERSION_OPTION  = -v
fprate=peak:
   OPTIMIZE           = -O2
   PASS1_OPTIMIZE     = -fprofile-generate
   PASS2_OPTIMIZE     = -fprofile-use

$ cat mat.sh
runcpu --config=mat | grep -e Training -e lbm.c -e .log
$ ./mat.sh 
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP -O2 -fprofile-generate lbm.c
Training 519.lbm_r with the train workload
gcc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP -O2 -fprofile-use      lbm.c
The log for this run is in /vampir/cpu2017/rc4/result/CPU2017.838.log
$  (Notes about examples) 

The PASSn_OPTIMIZE lines cause FDO to happen. The grep picks out a few lines of interest: compile using -fprofile-generate; run the training workload; recompile with -fprofile-use.

For much more detail, you can examine the log file. In this section, the log file from the above command is used (with light editing for readability, e.g. adjusted white space). Useful search strings are bolded.

'runcpu:' Search for runcpu: to verify that we have the correct log file:

$ go result                 (go)
/vampir/cpu2017/rc4/result
$ grep runcpu: CPU2017.837.log 
runcpu: runcpu --config=mat
$

Yes, this looks like the right log. Bring it up in an editor.

'Building' If you search for Building, you will find what was written to Makefile.deps (dependencies) and Makefile.spec (the actual Makefile). In this case, there are no dependencies.

  Building 519.lbm_r peak blue271: (build_peak_blue271.0000) [2017-02-14 05:26:09]
Wrote to makefile
'/vampir/cpu2017/rc4/benchspec/CPU/519.lbm_r/build/build_peak_blue271.0000/Makefile.deps':


# End dependencies
# These are the build dependencies
Wrote to makefile
'/vampir/cpu2017/rc4/benchspec/CPU/519.lbm_r/build/build_peak_blue271.0000/Makefile.spec':
TUNE=peak
LABEL=blue271
NUMBER=519
NAME=lbm_r
SOURCES= lbm.c main.c
EXEBASE=lbm_r
NEED_MATH=yes
BENCHLANG=C

BENCH_FLAGS      = -DSPEC_AUTO_SUPPRESS_OPENMP
CC               = gcc
CC_VERSION_OPTION = -v
OPTIMIZE         = -O2
OS               = unix
PASS1_OPTIMIZE   = -fprofile-generate
PASS2_OPTIMIZE   = -fprofile-use 

'specmake' Search for specmake. There will be several hits. For a more narrow search, use  ^specmake.*build  if your editor allows it. Below,

specmake clean removes old files
specmake build does the actual build
specmake options generates a summary list of options from the build.

Just after the specmake build is the first set of actual compile commands. There are several mandatory flags added by the toolset. For example, a flag is added to suppress OpenMP directives, because this is a SPECrate run. You can also see -O2 -fprofile-generate as requested in the config file.

Issuing make.clean command 'specmake --output-sync clean'
specmake --output-sync clean
Start make.clean command: 2017-02-14 05:26:09 (1487067969.8877)
rm -rf *.o  lbm.out
find . \( -name \*.o -o -name '*.fppized.f*' -o -name '*.i' -o -name '*.mod' \) -print | xargs rm -rf
rm -rf lbm_r
rm -rf lbm_r.exe
rm -rf core
Stop make.clean command: 2017-02-14 05:26:10 (1487067970.06632)
Elapsed time for make.clean command: 00:00:00 (0.178615808486938)
Issuing fdo_make_pass1 command 'specmake --output-sync --always-make build FDO=PASS1'
specmake --output-sync --always-make build FDO=PASS1
Start fdo_make_pass1 command: 2017-02-14 05:26:10 (1487067970.06816)
gcc -c -o lbm.o  -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP -O2 -fprofile-generate  lbm.c
gcc -c -o main.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP -O2 -fprofile-generate  main.c
gcc       -O2  -fprofile-generate        lbm.o main.o             -lm         -o lbm_r
Stop fdo_make_pass1 command: 2017-02-14 05:26:10 (1487067970.73817)
Elapsed time for fdo_make_pass1 command: 00:00:00 (0.670012950897217)
Issuing options1 command 'specmake --output-sync options FDO=PASS1'
specmake --output-sync options FDO=PASS1
Start options1 command: 2017-02-14 05:26:10 (1487067970.73965)

If you are curious about the build command

specmake --output-sync --always-make build FDO=PASS1

the details are:

specmake is GNU Make.
--output-sync consolidates compiler stdout and stderr messages by module
--always-make rebuilds all targets
build selects the target which is the benchmark executable in $SPEC/benchspec/Makefile.defaults
FDO=PASS1 turns on the Feedback Directed Optimization switches matching PASS1 in Makefile.defaults
For more information on how SPEC CPU 2017 builds work, see the Make Variables document.

'Training' Search for Training. At the top of this section, runcpu forces single threading (effectively disabling OpenMP) because this is a SPECrate run. It takes a bit over 26 seconds to run the training workload.

Training 519.lbm_r with the train workload
OpenMP environment variables removed: None
OpenMP environment variables in effect:
    OMP_NUM_THREADS       '1'
    OMP_THREAD_LIMIT      '1'
Pre-run environment changes:
    'OMP_NUM_THREADS' added: (value now '1')
    'OMP_THREAD_LIMIT' added: (value now '1')
Commands to run (specinvoke command file):
  -N C
  -C /vampir/cpu2017/rc4/benchspec/CPU/519.lbm_r/build/build_peak_blue271.0000
  -o lbm.out -e lbm.err ../build_peak_blue271.0000/lbm_r 300 reference.dat 0 1 100_100_130_cf_b.of 
Specinvoke: /vampir/cpu2017/rc4/bin/specinvoke -d /vampir/cpu2017/rc4/benchspec/CPU/519.lbm_r/build...
Issuing command '/vampir/cpu2017/rc4/bin/specinvoke -d /vampir/cpu2017/rc4/benchspec/CPU/519.lbm_r/...
/vampir/cpu2017/rc4/bin/specinvoke -d /vampir/cpu2017/rc4/benchspec/CPU/519.lbm_r/build/build_peak_...
Start command: 2017-02-14 05:26:11 (1487067971.19247)
Stop command: 2017-02-14 05:26:37 (1487067997.72349)
Elapsed time for command: 00:00:26 (26.5310192108154)
Workload elapsed time (copy 0 workload 1) = 26.366245 seconds
Copy 0 of 519.lbm_r (peak train) run 1 finished at 2017-02-14 05:26:37. Total elapsed time: 26.3662

Notice the section 'Commands to run'. The key line begins with -o which tells specinvoke to

send output to lbm.out (from -o lbm.out)
send errors to lbm.err (from -e lbm.err)
run the benchmark executable binary lbm_r (from ..path../lbm_r)
with arguments 300 reference.dat 0 1 100_100_130_cf_b.of

Some benchmarks run more than once and have multiple lines that start with -o in the corresponding location. For example, the compression benchmark 557.xz_r compresses several different kinds of input stream. You can see the inputs that SPEC has provided for training purposes in the directories nnn.benchmark/data/train/input and nnn.benchmark/data/all/input. In some cases, the training workloads required significant development effort. As a user of the suite you don't have to worry about that; you simply apply them.

About training fidelity: SPEC is aware that there is some variation in the fidelity between benchmark training workloads vs. the timed "ref" workloads. One might argue that a training data set is "not good enough": compilers might guess incorrectly if they rely on it; or argue that a set is "too good": compilers can guess too easily. In the real world also, not just in benchmarking, there is such variation, because training data sets can be difficult to find. If you would like to experiment with different training workloads, see the comments on using a sandbox in the document Avoiding runcpu; see the utility convert_to_development; and please be reminded that you must not represent measurements with other workloads as official SPEC metrics. [link to this paragraph]

'specmake' (again): If you search again for specmake, you will come to the second specmake build, which uses the generated profile.

Issuing fdo_make_pass2 command 'specmake --output-sync --always-make build FDO=PASS2'
specmake --output-sync --always-make build FDO=PASS2
Start fdo_make_pass2 command: 2017-02-14 05:26:38 (1487067998.25607)
gcc -c -o lbm.o  -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP -O2 -fprofile-use lbm.c
gcc -c -o main.o -DSPEC -DSPEC_CPU -DNDEBUG -DSPEC_AUTO_SUPPRESS_OPENMP -O2 -fprofile-use main.c
gcc       -O2  -fprofile-use        lbm.o main.o             -lm         -o lbm_r
Stop fdo_make_pass2 command: 2017-02-14 05:26:38 (1487067998.92049) 

And that's it. The tools did most of the work; the user simply set the PASSn flags in the config file.

VIII.D. Help, I've got too many log files!

If you do many builds and runs, you may find that your result directory gets too cluttered. Within a result directory, all output formats other than .rsf can be regenerated from your .rsf files. Therefore, you could reduce clutter by deleting HTML, PDF, and other reports. You can delete old .debug logs unless you plan to submit a support request.

Still feel cluttered? A simple solution is to move your result directory aside, giving it a new name. Don't worry about creating a new directory; runcpu will do so automatically. You should be careful to ensure no surprises for any currently-running users. If you move result directories, it is a good idea to also clean temporary directories at the same time.
Example:

cd $SPEC
mv result old-result
rm -Rf tmp/
cd output_root     # (If you use an output_root)
rm -Rf tmp/

Windows users: Windows users can achieve similar effects using the rename command to move directories, and the rd command to remove directories.

VIII.E. Finding the build directory

As described under "About Disk Usage" in runcpu.html, the CPU 2017 tools do the actual builds and runs in newly created directories. The benchmark sources are never modified in the src directory.

The build directories for a benchmark are located underneath that benchmarks' top-level directory, typically $SPEC/benchspec/CPU/nnn.benchmark/build (Unix) or %SPEC%\benchspec\CPU\nnn.benchmark\build (Windows).

(If you are using the output_root feature, then the first part of that path will change to be your requested root instead of SPEC.)

The build directories have logical names, typically of the form build_<tune>_<label>.0000. Continuing the FDO log example, this directory was created:

$ go lbm build                (go)
/vampir/cpu2017/rc4/benchspec/CPU/519.lbm_r/build
$ ls -ld *blue*
drwxrwxr-x  40 mat staff  1360 Feb 14 05:26 build_peak_blue271.0000
$ 

On Windows, you would say cd %SPEC%\benchspec\CPU\519.lbm_r\build followed by dir build*.

If the directory build_<tune>_<label>.0000 already exists when a new build is attempted for the same tuning and label, the directory will be re-used, unless:

• You have set the environment variable SPEC_CPU2017_NO_RUNDIR_DEL, or
• The directory has been left in a "locked" state by an aborted previous build.

In such cases, the 0000 will be incremented until a name is generated that is available. You can find locked directories by searching for lock=yes in the file $SPEC/benchspec/CPU/<nnn.benchmark>/run/list (Unix) or %SPEC%\benchspec\CPU\<nnn.benchmark>\run\list (Windows).

When more than one build directory has been created for a given tuning and label, you may need to trace the directory back to the specific build attempt that created it. You can do so by searching for the directory name in the log files:

$ grep build_peak_blue271.0000 *log | grep Building
CPU2017.838.log:  Building 519.lbm_r peak blue271: (build_peak_blue271.0000) [2017-02-14 05:26:09]
$  

In the above example, the grep command locates log #838 as the log that corresponds to this run directory. On Windows, of course, you would use findstr instead of grep.

VIII.F. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

VIII.G. For more information

For more information about how the run directories work, see the descriptions of specinvoke, specmake, and specdiff in utility.html.

IX. Power measurement

SPEC CPU 2017 v1.0 provided new capabilities for power measurement.
As of SPEC CPU 2017 v1.1, these capabilities are fully supported, and power results can be compared across systems.

Power measurement is optional.

Power measurement uses the same methods as other SPEC benchmarks, as initially introduced with the SPECpower_ssj®2008 benchmark suite.

IX.A. Getting started with power measurement

To get started:

1. Review the new SPEC CPU 2017 Run Rules regarding power, especially section 3.9 Power and Temperature Measurement.

2. You will need additional hardware:

   • An accepted power analyzer, which has been calibrated within the last year. See the list of accepted devices which is maintained by the SPECpower subcommittee.
   • An accepted temperature sensor (see the same list).

   • An x86-based Linux or Windows "Controller System", where you will run SPEC's Power and Temperature reporting tool, SPEC PTDaemon™. The PTDaemon controls your power analyzer and temperature sensor, translating vendor-specific command and hardware interfaces into a consistent interface for use by runcpu. Use the SPECpower Device Compatibility page to help you pick a controller system that is compatible with your power analyzer and temperature sensor.

3. On the controller system, follow the Power and Temperature Measurement Setup Guide to set up PTDaemon so that the power analyzer and the temperature meter are available at known network locations. There are also some examples just below that may help you.

4. On the system under test, edit your config file:

   • Enter the network locations into the power_analyzer and temp_meter fields. For example, if your controller system is named "regler", you might enter:

     power_analyzer = regler:8888
     temp_meter     = regler:8889

   • Set the voltage_range field.
   • Set the idle_current_range and the current range to initial approximations for the current that will be used. You may need to adjust these once you actually do some measurements. You also may find that you need to set multiple ranges.

5. Set power = yes in your config file, or use runcpu --power, and try a simple run, for example:

   $ runcpu -c myconfig --power --size=test --iterations=1 --tune=base --copies=1 519.lbm 

   You should see messages such as these:

   Connecting to power analyzers...
   Attempting to connect to PTD at regler:8888 with 30 second timeout...
   Connected to 1 power analyzers
   Connecting to temperature meters...
   Attempting to connect to PTD at regler:8889 with 30 second timeout...
   Connected to 1 temperature meters

IX.B. PTD Examples

Example connections: Your particular setup for PTDaemon will depend on the details of your power analyzer, temperature meter, and controller system. See the list of accepted devices and the SPECpower Device Compatibility page.

The diagram on the right shows one example setup, using a Linux controller system (named "regler") with a USB port connected to a temperature monitor, and another USB port connected to a power analyzer via a USB-to-RS232 converter. Your connections may differ. One copy of ptd controls the analyzer, making its measurements available at network location regler:8888. Another copy of ptd controls the temperature monitor and provides data to network location regler:8889. On the System Under Test (SUT), runcpu starts up a series of SPECrate jobs, and obtains power and temperature data from network locations regler:8888 and regler:8889. The temperature probe is hung about 25 mm (1 inch) from the primary air intake (as the rules require). All power to the SUT comes through the power analyzer.

Software example: Use the Power and Temperature Measurement Setup Guide to set up PTDaemon. The details will depend on your particular power analyzer, temperature sensor, and controller system; the following is only an example.

$ cd $SPEC
$ scp PTDaemon/binaries/ptd-linux-x86 root@regler:/root/ptd
root@regler's password:
ptd-linux-x86                100% 3778KB  16.8MB/s   00:00
$

$ lsusb | grep -v hub
Bus 001 Device 002: ID 046b:ff01 American Megatrends, Inc.
Bus 002 Device 002: ID 04b4:6560 Cypress Semiconductor Corp. CY7C65640 USB-2.0 "TetraHub"
Bus 005 Device 002: ID 046b:ff10 American Megatrends, Inc. Virtual Keyboard and Mouse
Bus 007 Device 002: ID 06cd:0121 Keyspan USA-19hs serial adapter
Bus 008 Device 002: ID 1608:0305 Inside Out Networks [hex] Watchport/H
$ ls /dev/*USB*
/dev/ttyUSB0  /dev/ttyUSB1
$ dmesg | grep ttyUSB
usb 7-2: Keyspan 1 port adapter converter now attached to ttyUSB0
usb 8-1: Edgeport TI 1 port adapter converter now attached to ttyUSB1
$ 

# cat start-ptd.sh
#!/bin/bash

PT=/root/ptd/ptd-linux-x86
LG=/root/ptd/logs

export LIBC_FATAL_STDERR_=1  # Send fatal errors to the output files

$PT    -l $LG/power.log -p 8888 wt210       /dev/ttyUSB0 >> $LG/power.out 2>&1 &
$PT -t -l $LG/temp.log  -p 8889 watchport/h /dev/ttyUSB1 >> $LG/temp.out  2>&1 &

# ./start-ptd.sh
# pgrep -l ptd
16980 ptd-linux-x86
16981 ptd-linux-x86
#

IX.C. Descriptive fields for power and temperature

The above will you get started. Prior to publishing any results, you will also need to fill out the various other config file fields that describe your power and temperature measurement for human readers, for example:
hw_power_{id}_cal_date
hw_power_{id}_connection
hw_temperature_{id}_connection
hw_temperature_{id}_setup.

There are many other such fields; please search for "power" and "temp" in the list of Reader Fields in the table of contents.

IX.D. Current ranges

Analyzer Power Ranges: Power analyzers typically need to be set according to the expected incoming power. In your config file, you estimate the current needed for the benchmarks, and then the SPEC toolset tells the analyzer what power range to use.

For example, if your power analyzer has three ranges of 1 Amp, 10 Amps, and 20 Amps, and your config file has:

intrate:
   current_range=7
fpspeed:
   current_range=12

then the middle range will automatically be selected when running integer rate, and the upper range will be selected when running floating point speed.

Requirements: The run rules require the tester to pick current ranges that meet certain conditions, notably including the condition that uncertainty must be reported at less than 1%. This section is intended to help you with current_range selection.

Auto-ranging: Some power analyzers can dynamically respond to incoming current and automatically pick a power range. Auto-ranging may be useful during initial setup, but must be avoided for reportable runs unless there is no other choice (in which case evidence must be preserved to prove that it was the only choice).

Initial Current Range: You will need to pick initial values for both the expected maximum current_range and the expected idle_current_range. There are several ways you can pick initial values:

• Use the manufacturer's specifications. Server manufacturers will often include a calculator on their website.
• Or, perhaps the service processor (aka "BMC", baseboard management controller) may tell you.

• Or, if your power analyzer supports auto-ranging, temporarily set the current_range=auto and do a single-iteration, base tuning, non-reportable run of floating point rate while exercising all available hardware threads. Then, either:

  • Observe the "Max. Power(W)" and "Idle Power(W)" from the reports and convert the watts to Amps (which will depend on your input voltage); or

  • Check out the PTD log, looking for the maximum Amps. For example, on a Linux controller system you could enter commands similar to these:

    # grep ",Amps" power.log | sort -nk8 -t, | tail -3
    Time,05-12-2019 07:41:48.300,Watts,2779.100000,Volts,209.010000,Amps,13.328000,PF,0.997600,Mark,507.cactuBSSN_r
    Time,05-12-2019 07:41:56.300,Watts,2783.300000,Volts,208.990000,Amps,13.349000,PF,0.997700,Mark,507.cactuBSSN_r
    Time,05-12-2019 07:41:47.300,Watts,2787.900000,Volts,208.980000,Amps,13.372000,PF,0.997700,Mark,507.cactuBSSN_r
    # grep "Mark,Idle" power.log | sort -nk8 -t, | tail -3
    Time,05-12-2019 02:06:45.600,Watts,1491.800000,Volts,210.440000,Amps,7.132000,PF,0.994000,Mark,Idle
    Time,05-12-2019 02:06:44.600,Watts,1492.600000,Volts,210.470000,Amps,7.135000,PF,0.994000,Mark,Idle
    Time,05-12-2019 02:06:47.600,Watts,1492.700000,Volts,210.430000,Amps,7.137000,PF,0.994000,Mark,Idle
    # 

    The above commands search the power log, sorting numerically by the 8th field, where fields are delimited by commas, and print the last few (highest) records. From the above example, one could set current_range=13.4 and idle_current_range=7.2.

  After you complete the non-reportable, single-iteration run, change the word "auto" to the number of Amps that were observed.

Adjusting the range: You may need to adjust your initial values after you do more runs. Keep an eye on the reported "Max. Power(W)" in the various reports. Watch for any reported errors. For example, if you see errors such as these:

ERROR: Average uncertainty of all samples from regler:8888 exceeds 1%
ERROR: Reading power analyzers returned errors

then you may have set current_range to a value that is too high.

On the other hand, if you see errors such as these:

Meter 'regler:8888' reports 10 errors
Meter 'regler:8888' reports 10 samples, of which only 0 are good
ERROR: Power analyzer 'regler:8888' returned no measurement uncertainty information

then one possiblity that you should check would be whether you may have set a current_range that is too low.

Multiple ranges: To meet the rules, you may need to pick different current ranges for different benchmarks, by setting current_range in benchmark-specific sections. For example, if you find that 507.cactuBSSN_r uses more current than other floating point rate benchmarks, perhaps you might set something like this:

fprate=base:
   current_range=9
507.cactuBSSN_r=base:
   current_range=13.4 

Note: it is allowed to set different current ranges for different benchmarks, even in base.

IX.E. Power questions?

If you run into difficulties, a few suggestions may be found in the FAQ.

Appendix A: Auxiliary Benchmark Sets

 # examples
  intrate:
     OPTIMIZE = -O2
  fprate:
     OPTIMIZE = -O3
  fpspeed:
     EXTRA_OPTIMIZE = -DSPEC_OPENMP -fopenmp
  default=peak:
     LDOPTIONS = --shared
         

This appendix describes the Auxiliary Benchmark Sets.
These sets are available but require caution. They are not recommended unless you are an expert user.

specspeed    intspeed plus fpspeed
specrate     intrate  plus fprate
cpu          all benchmarks: the same benchmarks as 'default'; higher priority than 'default'
openmp       benchmarks with OpenMP directives
serial_speed specspeed benchmarks without OpenMP
any_fortran  benchmarks using Fortran (in whole or in part)
pure_fortran benchmarks using Fortran and no other language
  .          Any other benchmark set from
  .             ls $SPEC/benchspec/CPU    (Unix)
  .             dir %SPEC%\benchspec\CPU  (Windows)

Considerations when using Auxiliary Benchmarks Sets:

• Do not try to use the preenv feature. It works only for the header section and these: default: intrate: intspeed: fprate: fpspeed:
• The base consistency rules must not be violated. Auxiliary sets that are supersets of the metrics are safe in that regard (specspeed: specrate: cpu:). Others may only be used in peak.

• Benchmarks are members of many auxiliary sets. (For example, 503.bwaves_r appears in sets specrate:, pure_fortran, and any_fortran among others). If a config file addresses more than one, conflicts are resolved alphabetically. Effectively, the precedence listed in Precedence Example 1 becomes:

    highest   named benchmark(s)
              suite name
              auxilary sets in alphabetical order
    lowest    default

Example: The diff command shows that only one line differs between two config files. benchmark_set.wrong.cfg falls into two traps.

$ diff --width 110 --side-by-side  --left-column \
>   benchmark_set.correct.cfg                          benchmark_set.wrong.cfg 
flagsurl       = $[top]/config/flags/gcc.xml          (
output_format  = text                                 (
output_root    = /tmp/benchmark_set                   (
runlist        = intspeed                             (
size           = test                                 (
default:                                              (
   CC          = gcc      -std=c99                    (
   CXX         = g++      -std=c++03                  (
   FC          = gfortran -std=f2003                  (
   # How to say "Show me your version"                (
   CC_VERSION_OPTION    = -v                          (
   CXX_VERSION_OPTION   = -v                          (
   FC_VERSION_OPTION    = -v                          (
default=base:                                         (
   OPTIMIZE             = -O1                         (
intspeed:  # correct                                  | openmp:    # wrong wrong wrong wrong wrong wrong
   EXTRA_OPTIMIZE       = -DSPEC_OPENMP -fopenmp      (
   preENV_OMP_STACKSIZE = 120M                        (
                                                      (
$

The script runs both in fake mode, and searches the output for the OMP_STACKSIZE setting:

$cat benchmark_set.sh 
runcpu --fakereportable --config=benchmark_set.correct | grep speed.txt
runcpu --fakereportable --config=benchmark_set.wrong   | grep speed.txt
cd /tmp/benchmark_set/result
grep STACKSIZE *txt
$
$ ./benchmark_set.sh 
    format: Text -> /tmp/benchmark_set/result/CPU2017.001.intspeed.txt
    format: Text -> /tmp/benchmark_set/result/CPU2017.002.intspeed.txt
CPU2017.001.intspeed.txt:    OMP_STACKSIZE = "120M"
$  (Notes about examples) 

Notice that only benchmark_set.correct.cfg gets the desired OMP_STACKSIZE. (The grep does not match CPU2017.002.intspeed.txt.)

More importantly, benchmark_set.correct.cfg compiles all C benchmarks the same way (as required). From CPU2017.001.intspeed.txt:

                              Base Optimization Flags
                              -----------------------
C benchmarks:
     -std=c99 -O1 -DSPEC_OPENMP -fopenmp
      

The consistency rule is violated by benchmark_set.wrong.cfg and results using it would be non-compliant. From CPU2017.002.intspeed.txt:

                              Base Optimization Flags
                              -----------------------
C benchmarks:
 600.perlbench_s: -std=c99 -O1
       602.gcc_s: Same as 600.perlbench_s
       605.mcf_s: Same as 600.perlbench_s
      625.x264_s: Same as 600.perlbench_s
        657.xz_s: -std=c99 -O1 -DSPEC_OPENMP -fopenmp

Appendix B. Troubleshooting

When something goes wrong, here are some things to check:

1. Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward looking for errors.

2. Did your desired switches get applied? Go to the build directory, and look at options*out.

3. Did the tools or your compilers report any errors? Look in the build directory at *out or *err.

4. What happens if you try the build by hand? See the section on specmake in utility.html.

5. If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html

6. Do you understand what is in your path, and why? Sometimes confusion can be greatly reduced by ensuring that you have only what is needed, avoiding, in particular, experimental and non-standard versions of standard utilities.

   Note: on Windows systems, SPEC recommends that Windows/Unix compatibility products should be removed from the %PATH% prior to invoking runcpu, in order to reduce the probability of certain difficult-to-diagnose error messages.

7. Try asking the tools to leave more clues behind, with keeptmp.

Appendix C. Obsolete/removed items

SPEC CPU® 2017 Config Files: Copyright © 2017-2020 Standard Performance Evaluation Corporation (SPEC®)