SPEC CPU2000 Config Files Last updated: 30-Oct-2001 jh

(To check for possible updates to this document, please see http://www.spec.org/cpu2000/docs/)

I. Introduction

This document assumes that you have already read:

• readme1st.htmland
• runspec.html

The runspec documents discuss the primary user interface for running SPEC CPU2000; with this document, attention turns more toward how things work inside.

A. What is a config file? (Background: benchmark philosophy.)

A config file contains:

• the instructions for building the benchmarks
• the instructions for running the benchmarks
• the description of the system under test
• the key to result reproducibility
• the repository for all information unique to a vendor/system
• the guts, the real controls, the gold, the Good Stuff
• the place where the benchmark philosophy hits the road

(Readers who just want to get on with the technical description should skip the remainder of this section; those who are interested in benchmark design philosophy should continue reading.)

A key decision that must be made by designers of a benchmark suite is whether to allow the benchmark source code to be changed when the suite is used.

If source code changes are allowed:

If source code changes are not allowed:

SPEC has chosen not to allow source code changes for the CPU2000 suite, except under very limited circumstances described below. By restricting source code changes, we separate the activity of porting benchmarks (which has a goal of being performance neutral), from the activity of using the benchmarks (where the goal is not neutrality, it's Get The Best Score You Can.) Prior to the first use of CPU2000, SPEC therefore invested substantial effort to port the suite to as many platforms as practical. Testing was done on 7 different hardware architectures, 11 versions of Unix (including 4 versions of Linux) and 2 versions of NT.

Are source code changes ever allowed? Normally, no. But if you discover a reason why you believe such a change is required, SPEC wants to hear about it, and will consider such requests for a future revision of the suite. SPEC will normally not accept publication of CPU2000 results using modified source code, unless such modifications are unavoidable for the target environment, are submitted to SPEC, are made available to all users of the suite, and are formally approved by a vote.

So, if source code changes are not allowed, but the benchmarks must be compiled in a wide variety of environments, can the users at least write their own Makefiles, and select -D options to select different environments? The answer to these two questions are "no", and "yes", respectively:

• No, you can't write your own Makefiles.
• But yes, you can select portability options (such as -D ).

You do this in the config file, which contains a centralized collection of all the portability options and optimization options for all the benchmarks in a CPU2000 suite. The SPEC tools then automatically generate the Makefiles for you.

The config file contains places where you can specify the characteristics of both your compile time and run time environments. It allows the advanced user to perform detailed manipulation of Makefile options, but retains all the changes in one place so that they can be examined and reproduced.

The config file is one of the key ingredients in making results reproducible. For example, if a customer would like to run the CPU2000 suite on her own SuperHero Model 4 and discover how close results are in her environment to the environment used when the vendor published a CPU2000 result, she should be able to do that using only 3 ingredients:

1. The SPEC CPU2000 CD ordered from SPEC
2. The software and hardware as specified in the full disclosure when the vendor submitted the original results
3. The config file from the vendor's submission

B. What does a config file affect?

A config file contains directives that are addressed to four different consumers: runspec , specmake, the shell, and the reader of the results:

1. runspec

Various aspects of the operation of runspec can be set within the config file. For example, if michael.cfg includes the lines:

   output_format = asc,ps
   tune          = base
   reportable    = 1
   runlist       = fp

then the defaults for the runspec command would change as specified. A user who types either of the following two commands would get precisely the same effect:

   runspec --config=michael
   runspec --config=michael --output=asc,ps --tune=base --reportable fp

2. specmake

The tool specmake is simply gnu make renamed to avoid any possible conflicts with other versions of make that may be on your system. Config file lines such as:

   CC           = cc
   CPORTABILITY = -DSPEC_CPU2000_LP64
   OPTIMIZE     = -O4

are written to the Makefile set that is ultimately used to build the benchmark, and these lines are interpreted by specmake .

3. The shell

Certain commands are interpreted by the shell, for example:

   fdo_pre0     = mkdir /tmp/ahmad; rm -f /tmp/ahmad/*

Runspec passes the above command to the shell prior to running a training run for feedback directed optimization, and the shell actually carries out the requested commands. It is therefore important to peruse a config file you are given before using it.

4. The reader of the results

A SPEC CPU2000 result, when submitted to http://www.spec.org/, is expected to contain all the information needed so that readers can understand exactly what was tested. Config file lines such as these are addressed to that reader:

   test_date = Nov-1999
   hw_avail  = Apr-1999
   sw_avail  = May-1999
   notes015  = Note: Feedback directed optimization was not used

In addition, the config file itself is available to the reader at http://www.spec.org/. The config file is presented in its entirety, with one exception described below under Comments and whitespace. The reason that the config file is made available is because it is so important to reproducing results, as described in the Introduction. The config file is saved on every run, as a compressed portion of the rawfile.

C. Config file structure

1. Comments and whitespace

Comment lines begin with a # , and can be placed anywhere in a config file. When the config file is saved (as an encoded portion of the rawfile), the comments are included. But if a comment line begins with #> , it will not be saved in the rawfile. Thus you could use # for most of your comments, and use #> for proprietary information, such as:

   #> I didn't use the C++ beta test because of Bob's big back-end bug.

Blank lines can be placed anywhere in a config file.

Spaces within a line are almost always ignored. Of course, you wouldn't want to spell OPTIMIZE as OPT I MIZE, but you are perfectly welcome to do either of the following:

   OPTIMIZE=-O2
   OPTIMIZE   =  -02

The one place where spaces are considered significant is in notes, where the tools assume you are trying to line up your comments in the full disclosure reports.

2. Header section

A config file contains the following:

• A header section
• An arbitrary number of named sections, delimited by section markers (described below).
• An MD5 section

The header section is simply the first section, prior to the first occurence of a named section.

Most attempts to address runspec itself must be done in the header section. For example, if you want to set reportable=1 , you must do so before any occurrences of section markers.

The next several pages are devoted to sections that you name; the MD5 section is explained after that.

3. Named Sections

A "section marker" is a four-part string of the form:

   benchmark=tuning=extension=machine:

These are referred to below as the 4 "section specifiers". The allowed values for the section specifiers are:

Trailing default sections may be omitted. Thus all of the following are equivalent:

   252.eon=base=default=default:
   252.eon=base=default:
   252.eon=base:

Section markers can be entered in any order. Section markers can be repeated; material from identical section markers will automatically be consolidated. That is, you are welcome to start one section, start a different one, then go back and add more material to the first section. But please note that since there is no marker for the header section, you cannot go back to it.

a. Precedence for the benchmark specifier

By constructing section markers, you specify how you would like your options applied, with powerful defaulting and over-riding capabilities. Let's examine these capabilities by way of examples. We start with a simple config file:

   % cat tmp.cfg
   runlist       = swim
   size          = test
   iterations    = 1
   tune          = base
   output_format = asc
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -O2

   %runspec --config=tmp | grep swim.f
   f90 -c -o swim.o            -O2   swim.f
   %

The config file above is designed for quick, simple testing: it runs only one benchmark, namely 171.swim , using the smallest ( test) workload, runs it only once, uses only base tuning, outputs only the text-format (ASCII) report, and displays the build commands to the screen. To use it, we issue a runspec command, and pipe the output to grepto search for the actual generated compile command. (Alternatively, on NT, we could use findstr on the generated log file). And, indeed, the tuning applied was the expected -O2 .

Now consider this example, where a section marker is added that has the first specifier set to fp , for the floating point suite:

   % cat tmp.cfg
   runlist       = swim
   size          = test
   iterations    = 1
   tune          = base
   output_format = asc
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -O2

   fp=default=default=default:
   OPTIMIZE = -O3

   % runspec --config=tmp | grep swim.f
   f90 -c -o swim.o            -O3   swim.f
   %

The second OPTIMIZE command is used above because the reference to the floating point suite is considered to be more specific than the overall default.

Furthermore, we can add a specifier that mentions swim by name:

   % cat tmp.cfg
   runlist       = swim
   size          = test
   iterations    = 1
   tune          = base
   output_format = asc
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -O2

   fp=default=default=default:
   OPTIMIZE = -O3

   171.swim=default=default=default:
   OPTIMIZE = -O4

   % runspec --config=tmp | grep swim.f
   f90 -c -o swim.o            -O4   swim.f
   %

The third OPTIMIZE command wins above, because it is included in the section that is considered to be the most specific. But what if we had said these in a different order?

   % cat tmp.cfg
   runlist       = swim
   size          = test
   iterations    = 1
   tune          = base
   output_format = asc
   teeout        = 1

   171.swim=default=default=default:
   OPTIMIZE = -O4

   default=default=default=default:
   OPTIMIZE = -O2

   fp=default=default=default:
   OPTIMIZE = -O3

   % runspec --config=tmp | grep swim.f
   f90 -c -o swim.o            -O4   swim.f
   %

Notice above that the order of entry is not significant; it's the order of precedence from least specific to most specific.

Note: when a specifier is listed more than once at the same descriptive level, the last instance of the specifier is used. Consider this case:

   171.swim=default=default=default:
   OPTIMIZE = -O4

   171.swim=default=default=default:
   OPTIMIZE = -O3

The ending value of OPTIMIZE for 171.swim is -O3, not -O4 .

b. Precedence for the tuning specifier

The tuning specifier is the the second in the group of four, taking the value "default", "base", or "peak". Here is an example of its use:

   % cat tmp.cfg
   runlist       = swim
   size          = test
   iterations    = 1
   tune          = base,peak
   output_format = asc
   teeout        = 1

   default=default=default=default:
   FC       = f90
   F77      = f77

   default=base=default=default:
   FC       = kf90

   default=peak=default=default:
   F77      = kf77

   % runspec --config=tmp | grep swim.f | grep swim.o
   kf90 -c -o swim.o                swim.f
   kf77 -c -o swim.o                swim.f
   %

In the above example, we compile swim twice: once for base tuning, and once for peak. Notice that in both cases the compilers defined by the more specific section markers have been used, namely kf90 and kf77, rather than the f90 and f77 from default=default=default=default.

(The alert reader may ask here "Why does FC apply to base and F77 apply to peak?" The reason for this is that in base, the CPU2000 Run Rules allow only one Fortran compiler, which must be used for both the benchmarks that use the older Fortran-77 features, and for the benchmarks that use the newer Fortran-90 features. But for peak tuning your definition of FC is used for source files *.f90, and your definition of F77 is used for source files *.f .)

c. Precedence for the extension specifier

Now let's examine the third section specifier, extension:

   % cat tmp.cfg
   runlist       = swim
   size          = test
   iterations    = 1
   tune          = base
   output_format = asc
   teeout        = 1

   default=default=default=default:
   LIBS = -lm

   default=default=ev6=default:
   LIBS = -lm_ev6

   % runspec --config=tmp | grep swim.o
   f90 -c -o swim.o                swim.f
   f90         swim.o  -lm   -o swim
   % runspec --config=tmp --extension=ev6 | grep swim.o
   f90 -c -o swim.o                swim.f
   f90         swim.o  -lm_ev6   -o swim
   %

   % cd $SPEC/benchspec/CFP2000/171.swim/exe
   % ls -lt | head -3
   total 3400
   -rwxr-xr-x   1 john     users      65536 Dec 10 02:47 swim_base.ev6
   -rwxr-xr-x   1 john     users      49152 Dec 10 02:47 swim_base.none
   %

Notice above that two different versions of swim were built from the same config file, and both executables are present in the exe directory for swim.

d. Combining specifier types

What if more than one section applies to a particular benchmark? Consider this example:

   % cat tmp.cfg
   runlist       = swim
   size          = test
   iterations    = 1
   tune          = peak
   output_format = asc
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -O2
   F77      = f77
   LIBS     = -lm

   171.swim=default=default=default:
   OPTIMIZE = -O4

   default=peak=default=default:
   F77      = kf77

   default=default=ev6=default:
   LIBS = -lm_ev6

   % runspec --config=tmp --extension=ev6 | grep swim.o
   kf77 -c -o swim.o            -O4   swim.f
   kf77     -O4   swim.o  -lm_ev6   -o swim
   %

Notice above that all three sections applied: the section specifier for 171.swim, the specifier for peak tuning, and the specifier for extension ev6.

e. Precedence among section types

If sections contradict each other, the order of precedence is:

     highest     benchmark
                 suite
                 tuning
     lowest      extension

And this order can be demonstrated as follows:

   % cat tmp.cfg
   size          = test
   iterations    = 1
   output_format = asc
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -O0
   fp=default=default=default:
   OPTIMIZE = -O1
   171.swim=default=default=default:
   OPTIMIZE = -O3
   default=peak=default=default:
   OPTIMIZE = -O4
   default=default=ev6=default:
   OPTIMIZE = -O5

   % runspec --conf=tmp swim | grep swim.f
[1]  f90 -c -o swim.o            -O3   swim.f
   % runspec --conf=tmp --tune=peak swim | grep swim.f
[2]  f77 -c -o swim.o            -O3   swim.f
   % runspec --conf=tmp --extension=ev6 swim | grep swim.f
[3]  f90 -c -o swim.o            -O3   swim.f
   %
   % runspec --conf=tmp --tune=base mgrid | grep mgrid.f
[4]  f90 -c -o mgrid.o            -O1   mgrid.f
   % runspec --conf=tmp --tune=peak mgrid | grep mgrid.f
[5]  f77 -c -o mgrid.o            -O1   mgrid.f
   % runspec --conf=tmp --extension=ev6 mgrid | grep mgrid.f
[6]  f90 -c -o mgrid.o            -O1   mgrid.f
   %
   % runspec --conf=tmp --tune=base gzip | grep gzip.c
[7]  cc -c -o gzip.o            -O0   gzip.c
   % runspec --conf=tmp --tune=peak gzip | grep gzip.c
[8]  cc -c -o gzip.o            -O4   gzip.c
   % runspec --conf=tmp --extension=ev6 gzip | grep gzip.c
[9]  cc -c -o gzip.o            -O5   gzip.c
   % runspec --conf=tmp --tune=peak --extension=ev6 gzip | grep gzip.c
[10] cc -c -o gzip.o            -O4   gzip.c

Notice above that the named benchmark always wins: lines [1], [2], and [3]. If there is no section specifier that names a benchmark, but there is a section specifier that names a suite, then the suite wins: lines [4], [5], and [6]. If there are no applicable benchmark or suite specifiers, then tuning or extension can be applied: lines [8] and [9]. But if both tuning and extension are applied, tuning wins [10].

4. MD5

The final section of your config file is generated automatically by the tools at compile time, and looks something like this:

   __MD5__
   168.wupwise=peak=nov14a=default:
   # Last updated Sun Nov 14 04:49:15 1999
   optmd5=6cfebb629cf395d958755f68a6c9e05b
   exemd5=3d51c75d1c97918bf212c4fb47a30003

   183.equake=peak=nov14a=default:
   # Last updated Sun Nov 14 04:49:22 1999
   optmd5=f3f0d2022f2be4a9530c000b1e883008
   exemd5=923a7bfd6f68ccf0d6e49261fa1c2030

The "MD5" is a checksum that ensures that the binaries referenced in the config file are in fact built using the options described therein. For example, if you edit the config file to change the optimization level for 255.vortex, the next time the file is used the tools will notice the change and will recompile vortex.

You can optionally disable this behavior, but doing so is strongly discouraged. See the sarcastic remarks in the description of check_md5 , below.

If you would like to see what portions of your config file are used in computing the MD5 hash, runspec with --debug=30 or higher, and examine the log file.

5. Variable Substitution

You can do variable substitution using your config file. But, as described in the Introduction, a config file can affect runspec itself, specmake, and the shell. Therefore, effective use of variable substitution requires you to be aware of which software is doing the substitution:

• If runspec itself does the substitution, it uses perl variable interpolation. The items to be substituted usually look something like this: $hw_avail.
• If runspec generates an external command which is handed off to your shell (or command interpreter), the variable will be interpreted according to the rules of your shell. Such a variable might look like this: \$SPEC.
• If the variable is handed off to specmake , it is interpreted by the usual rules of gnu make. Such variables might look like this: $(FOPTIMIZE).

a. By runspec

Substitution by runspec itself uses perl variable interpolation. Only perl scalars (denoted by a leading $) can be interpolated. For example, notes001 below uses the log file number (generated by the tools) and the hardware availability date (which was set directly):

   % cat tmp.cfg
   size          = test
   iterations    = 1
   output_format = asc
   teeout        = 1
   expand_notes  = 1
   runlist       = swim
   tune          = base
   hw_avail      = May-2000
   notes001      = This run is from log.$lognum with hw_avail $hw_avail

   % runspec -c tmp | grep asc
   Identifying output formats...asc...config...html...pdf...ps...raw...
      format: ASCII -> /greg/result/CFP2000.101.asc
   % grep with /greg/result/CFP2000.101.asc
      This run is from log.101 with hw_avail May-2000

As another example, let's say you want to go for minimal performance. You might want to do this with the 'nice' command. You can say:

   submit=nice 20 '$command'

and the $command gets expanded to whatever would normally be executed but with 'nice 20' stuck in front of it.

If you'd like a complete list of the variables that you can use in your commands (relative to the config file you're using), set runspec 's verbosity to 80 or higher (-v 80) and either do a run that causes a command substitution to happen, or run with expand_notes set to 1.

To put text immediately after a variable, you need to make it possible for the parser to see the variable that you want, by using braces:

   % tail -2 tmp.cfg
   notes001 =You have done ${lognum}x runs tonight, aren't you tired yet?

   % runspec -c tmp | grep asc
   Identifying output formats...asc...config...html...pdf...ps...raw...
      format: ASCII -> /john/result/CFP2000.103.asc
   % grep done /john/result/CFP2000.103.asc
   You have done 103x runs tonight, aren't you tired yet?

(1) Example: cleaning files before a training run

Perhaps a more useful example is this one, which directs the shell to clean files related to the current executable in the temporary directory, before a training run for feedback directed optimization:

   fdo_pre0     = mkdir /tmp/pb; rm -f /tmp/pb/${baseexe}*

NOTICE in this example that although the commands are carried out by the shell, the variable substitution is done by runspec .

(2) Example: submitting to multiple nodes, Compaq Tru64 Unix

The following lines submit to multiple nodes on a Compaq AlphaServer SC Series Supercomputer running Tru64 Unix:

   submit= echo "$command" > dobmk; prun -n 1 sh dobmk
   command_add_redirect=1

In this example, the command that actually runs the benchmark is written to a small file, "dobmk", which is then submitted to a remote node selected by "prun". (The parallel run command, prun, can execute multiple copies of a process, but in this case we have requested just one copy by saying "-n 1". The SPEC tools will create as many copies as required.)

The "command_add_redirect" is crucial. What happens without it?

   % head -1 $SPEC/config/tmp1.cfg
   submit= echo "$command" > dobmk; prun -n 1 sh dobmk
   % runspec --config=tmp1 --size=test -n 1 --rate --users=1 swim
   ...
   % cat $SPEC/benchspec/CFP2000/171.swim/run/00000006/dobmk
   ../00000006/swim_base.none

Now let's use command_add_redirect, and see how dobmk changes:

   % diff $SPEC/config/tmp1.cfg $SPEC/config/tmp2.cfg
   1a2
   > command_add_redirect=1
   % runspec --config=tmp2 --size=test -n 1 --rate --users=1 swim
   ...
   % cat $SPEC/benchspec/CFP2000/171.swim/run/00000006/dobmk
   ../00000006/swim_base.none < swim.in > swim.out 2> swim.err

Notice that with command_add_redirect=1, the substitution for $command includes both the name of the executable and the file assignments for standard in, standard out, and standard error. This is needed because otherwise the files would not be connected to swim on the remote node. That is, the former generates [*]:

   echo "swim_base.none" > dobmk; 
   prun -n 1 sh dobmk < swim.in > swim.out 2> swim.err

And the latter generates [*]:

   echo "swim_base.none < swim.in > swim.out 2> swim.err" > dobmk; 
   prun -n 1 sh dobmk

b. By the shell

(1) Protecting shell variables

Because perl variables look a lot like shell variables, you need to specially protect shell variables if you want to prevent perl from trying to interpret them. Notice what happens with the protected and unprotected versions:

   % cat tmp.cfg
   size                 = test
   iterations           = 1
   output_format        = asc
   teeout               = 1
   expand_notes         = 1
   runlist              = swim
   tune                 = base,peak
   use_submit_for_speed = 1

   default=peak=default=default:
   submit = echo "home=$HOME; spec=$SPEC;" >/tmp/chan; $command

   default=base=default=default:
   submit = echo "home=\$HOME; spec=\$SPEC;" > /tmp/nui; $command

   % runspec --config=tmp > /dev/null
   % cat /tmp/chan
   home=; spec=;
   % cat /tmp/nui
   home=/usr/users/chris; spec=/cpu2000;
   %

Please note that, by default, 'submit' is only applied to rate runs. In this example, we used it for a speed run as well, by setting

   use_submit_for_speed = 1

in the configuration file.

(2) Example: submitting to multiple nodes, SGI Origin 2000

Here is a more complex example which uses substitution by both runspec and by the shell (line wraps added for readability):

   submit= 
      let "MYNUM=$SPECUSERNUM" ; 
      let "NODE=\$MYNUM/2"; 
      export NODE=/hw/nodenum/\$NODE; 
      let "CPU=2*((\$MYNUM+1)/2)-\$MYNUM "; 
      export CPU; 
      /usr/sbin/dplace -place \$SPEC/submit.pf -mustrun $command

Runspec substitutes the current user number for $SPECUSERNUM , and then passes the above command to the shell which does the substitutions for CPU and NODE. This example originated on an SGI Origin 2000 system, where there are two CPUs per node. Suppose that runspec is about to submit the copy for user number 17. In that case:

   submit= 
      let "MYNUM=$SPECUSERNUM" ;             # i.e. 17
      let "NODE=\$MYNUM/2";                  # So, NODE=8
      export NODE=/hw/nodenum/\$NODE;        # Now NODE=/hw/nodenum/8
      let "CPU=2*((\$MYNUM+1)/2)-\$MYNUM ";  # and CPU=1
      export CPU; 
      /usr/sbin/dplace                       # So we execute dplace
        -place \$SPEC/submit.pf              # using $SPEC/submit.pf
        -mustrun $command                    # and the expected command

The desired $command is run under the control of submit.pf, with $NODE=/hw/nodenum/8 and $CPU=1. Here are the contents of the placement file, submit.pf:

   memories 1 in topology physical near $NODE
   threads 1
   run thread 0 on memory 0 using cpu $CPU

Two important notes about the above submit command:

1. $SPECUSERNUM must be spelled exactly as it is in the examples above. It is a limitation of the tools that saying things like ${SPECUSERNUM} just won't work.
2. The above example was constructed using the first release of the SPEC CPU2000 suite. It should still work with the V1.1 update, but could probably be simplified. In V1.0, only the first occurrence of $SPECUSERNUM was substituted. That's why the example above immediately assigns the result to MYNUM , which can then be used repeatedly.

c. By specmake

For an example of variable substitution handled by specmake , see

   $SPEC/docs/example-advanced.cfg
   %SPEC%\docs.nt\example-advanced.cfg

Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake .

d. Limitations on variable substitution

Once runspec hands control over to specmakeor to the shell, the results of further substitution are invisible to runspec . For this reason, you can't say:

   wrong:MYDIR = /usr/paul/compilers
      F90   = $(MYDIR)/f90
      notes001 = compiler: $(F90)

6. Shell-like "here documents" and continued lines

Shell-style "here documents" are supported for setting variables to multi-line values. Continued lines (with \) are also supported:

   $ cat tmp2.cfg
   expand_notes=1

   foo = <<EOT
   This +
   is a +
   test +
   EOT

   bar = \
         and +\
         so  +\
         is  +\
         this+\
   notes01 = $foo
   notes02 = $bar
   $ runspec --config=tmp2 --size=test --iterat=1 swim | grep asc
   Identifying output formats...asc...config...html...pdf...ps...raw...
      format: ASCII -> /cpu2000/result/CFP2000.024.asc
   $ grep + ../result/*24.asc
      This +
      is a +
      test +
            and +
            so  +
            is  +
            this+

Note: although continued lines are supported, they are rarely used. The more common method of continuation is by appending a number to a field, as described in the section "Field scoping and continuation".

7. Included files

It is possible to include another file in your config file. A typical use for this feature might be to keep all the software information in the main config file, but to include the hardware information about the current System Under Test (SUT) in another file. For example:

   % cat tmp.cfg
   sw_compiler = GoFast C/C++ V4.2
   sw_avail    = Mar-2000
   include: SUT.inc

   default=default=default=default:
   OPTIMIZE = -O4

   % cat SUT.inc
   hw_model = SuperHero IV
   hw_avail = Feb-2000

   % runspec --config=tmp --iterations=1 --size=test swim

After the above command is completed, we end up with a report that mentions both hardware and software dates:

      ... Hardware availability: Feb-2000
      ... Software availability: Mar-2000

II. Config file options for runspec

This section documents options that control the operation of runspec itself.

A. Options for runspec that can also be used on the command line

The following items can be specified in a config file, and have the same meaning as if they are specified on the runspec command line. Please see runspec.htmlfor details. If an item is specified in both places, the command line wins.

All of these should be specified in your config file header section, unless the description states that the item can be applied to individual benchmarks.

B. Options for runspec that can only be used in a config file

The following options control the operation of runspec , but can not be specified on the command line. Instead, they must be specified in the config file.

All of these should be specified in your config file header section, unless the description states that the item can be applied to individual benchmarks.

III. Config file options for specmake

For a complete list of options that are used by specmake , please see $SPEC/docs/makevars.txt and notice which variables are documented as OK to mention in a config file.

Here are the commonly used variables:

Note that you can also make up your own variable names, which specmake will use (and perform substitution on). For an example of this feature, see $SPEC/docs/example-advanced.cfg.

IV. Config file options for the shell

The following config file options will cause their contents to be executed by the shell (or the Windows/NT command interpreter):

For an example of one of the fdo_ options, see the "Log file example" later in this document.

V. Config file options for the reader

Whether or not you submit your result to SPEC, you should fully disclose how you achieved the result. If it requires the installation of the GoFastLinker, you should say so. By setting the appropriate fields in the config file, you can cause information about the GoFastLinker to appear in all reports.

A. Pre-defined fields

Here are the pre-defined fields that you can set:

B. Field scoping and continuation

Note that these fields are scoped by section headers - that is, you can vary them according to the type of run that you are doing. You can also continue any of these fields to another line by appending a numeral to it. Here is an example of both these features:

   default=default=default=default:
   sw_compiler  = Compaq C X6.2-259-449AT 
   CC           = cc -v      

   int=default=default=default:  
   sw_compiler2 = DIGITAL C++ V6.1-029-408B6
   CXX          = cxx -v

   fp=default=default=default:
   sw_compiler2 = Compaq Fortran V5.3
   sw_compiler3 = KAP Fortran V4.2
   FC           = kf90 -v

Notice in the above example that the information about the C compiler will be printed for both integer and floating point runs. The information about the C++ compiler will be printed only for integer runs; the information about Fortran compilers will be printed only for floating point runs.

C. Free form notes

In addition to the pre-defined fields, you can write as many free-form notes as you wish. Start your notes with the characters "notes", and then add numbers and/or underscores. The notes will be sorted before being printed. For example:

   % cat tmp.cfg
   size                 = test
   iterations           = 1
   output_format        = asc
   teeout               = 1
   runlist              = swim
   tune                 = base

   notes01_1 = ++ how
   notes02   = ++ you?
   notes01_2 = ++ are
   notes01   = ++ Alex,
   notes000  = ++ hi

   default=base=default=default:

   % runspec --config=tmp > /nev/dull
   % ls -t ../result/*asc | head -1
   ../result/CFP2000.111.asc
   % grep ++ ../result/CFP2000.111.asc
      ++ hi
      ++ Alex,
      ++ how
      ++ are
      ++ you?
   %

You can also use notes to describe software or hardware information with more detail beyond the predefined fields. For an example of where this might be useful, see $SPEC/docs/example-medium.cfg and search for "patch".

VI. Files output during a build

A. Automatic backup of config files

It was mentioned above that the MD5 section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

   % ls /home/jim/CPU2000/config/tmp.c*
        tmp.cfg             tmp.cfg.19991210aq  tmp.cfg.19991210j
        tmp.cfg.19991210    tmp.cfg.19991210ar  tmp.cfg.19991210k
        tmp.cfg.19991210a   tmp.cfg.19991210as  tmp.cfg.19991210l
        tmp.cfg.19991210aa  tmp.cfg.19991210at  tmp.cfg.19991210m
        tmp.cfg.19991210ab  tmp.cfg.19991210au  tmp.cfg.19991210n
        tmp.cfg.19991210ac  tmp.cfg.19991210av  tmp.cfg.19991210o
        tmp.cfg.19991210ad  tmp.cfg.19991210aw  tmp.cfg.19991210p
        tmp.cfg.19991210ae  tmp.cfg.19991210ax  tmp.cfg.19991210q
        tmp.cfg.19991210af  tmp.cfg.19991210ay  tmp.cfg.19991210r
        tmp.cfg.19991210ag  tmp.cfg.19991210az  tmp.cfg.19991210s
        tmp.cfg.19991210ah  tmp.cfg.19991210b   tmp.cfg.19991210t
        tmp.cfg.19991210ai  tmp.cfg.19991210ba  tmp.cfg.19991210u
        tmp.cfg.19991210aj  tmp.cfg.19991210c   tmp.cfg.19991210v
        tmp.cfg.19991210ak  tmp.cfg.19991210d   tmp.cfg.19991210w
        tmp.cfg.19991210al  tmp.cfg.19991210e   tmp.cfg.19991210x
        tmp.cfg.19991210am  tmp.cfg.19991210f   tmp.cfg.19991210y
        tmp.cfg.19991210an  tmp.cfg.19991210g   tmp.cfg.19991210z
        tmp.cfg.19991210ao  tmp.cfg.19991210h
        tmp.cfg.19991210ap  tmp.cfg.19991210i

If this feels like too much clutter, you can disable the backup mechanism, as described under backup_config . Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove the clutter, for example by typing:

   rm *.cfg.199912*

B. The log file and verbosity levels

$SPEC/result (Unix) or %SPEC%\result (NT) contains reports and log files. When you are doing a build, you will probably find that you want to pay close attention to the log file. Depending on the verbosity level that you have selected, it will contain enormous amounts of information about how your build went.

The CPU2000 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information almost certainly worthless to anyone not developing the tools themselves (level 99). Note: selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

The 'level' referred to in the table below is selected either in the config file verbose option or in the runspec command as in 'runspec --verbose n '.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 3. This means that on your screen you will get messages at levels 0 through 3, and 100 through 103. Additionally, in your log file, you'll find messages at levels 104 through 199.

C. Log file example: Feedback-directed optimization.

Let's consider the log file for a compile with feedback-directed optimization (FDO). This will serve both as an example of how you accomplish such a compile and how to interpret a log file. [Note: this section uses an actual log file, but white space has been liberally adjusted, and much extraneous output has been removed.]

Feedback-directed optimization typically means that we want to compile a program twice: the first compile creates an image with certain instrumentation. Then, we run the program, and data about that run is collected (a profile). When the program is re-compiled, the collected data is used to improve the optimization.

The log file tells us what is written to the file Makefile.spec, along with lots of state information. It tells us that our config file set certain options:

   -----------------------------------
     Building 197.parser ref base nov14a default
     Wrote to makefile 
   '/cpu2000/benchspec/CINT2000/197.parser/run/00000002/Makefile.spec':
   
   PASS1_CFLAGS     = -prof_gen_noopt    -prof_dir /tmp/pb
   PASS2_CFLAGS     = -prof_use_feedback -prof_dir /tmp/pb
   baseexe          = parser
   fdo_pre0         = mkdir /tmp/pb; rm -f /tmp/pb/${baseexe}*
   feedback         = 1

To tell the tools that we want to use FDO, we simply set PASSn_CFLAGS . If the tools see any use of PASSn_xxxxx , they will perform multiple compiles. The particular compiler used in this example expects to be invoked twice: once with -prof_gen_noopt and then again with -prof_use_feedback. Note that we have also requested special processing before we start, in the fdo_pre0 step. The variable ${baseexe} is substituted with the name of the generated executable, minus any extensions or directories:

   Issuing fdo_pre0 command 'mkdir /tmp/pb; rm -f /tmp/pb/parser* 

Below, the first compile is done. Notice that specmake is invoked with FDO=PASS1 , which causes the switches from PASS1_CFLAGS to be used. (If you want to understand exactly how this affects the build, read $SPEC/benchspec/Makefile.defaults , along with the document $SPEC/docs/makevars.txt .)

   Output from fdo_make_pass1 
      'specmake  FDO=PASS1 build > fdo_make_pass1.out 2> fdo_make_pass1.err':
   cc -v  -prof_gen_noopt  -prof_dir /tmp/pb  -DSPEC_CPU2000  -v -arch ev6 
      -fast analyze-linkage.c and.c build-disjuncts.c extract-links.c ...

The next section shows how specinvoke runs the benchmark for the training run, according to the commands in speccmds.cmd. Specinvoke executes the instrumented parser using the training data set as input. (For more information on specinvoke, see utility.html.)

   Training 197.parser
   Commands to run: 
   -u /cpu2000/benchspec/CINT2000/197.parser/run/00000002
   -i train.in -o train.out -e train.err ../00000002/parser 2.1.dict -batch

   Specinvoke: /cpu2000/bin/specinvoke -d 
      /cpu2000/benchspec/CINT2000/197.parser/run/00000002 
      -e speccmds.err -o speccmds.out -f speccmds.cmd

Finally, the compiler is run a second time, this time to use the profile feedback and build a new executable. Notice that this time, specmake is invoked with FDO=PASS2 , which is why the compile picks up the PASS2_CFLAGS:

   Output from fdo_make_pass2 
   'specmake  FDO=PASS2 build > fdo_make_pass2.out 2> fdo_make_pass2.err':
   cc -v  -prof_use_feedback -prof_dir /tmp/pb -DSPEC_CPU2000  -v -arch ev6  
      -fast analyze-linkage.c and.c build-disjuncts.c extract-links.c ...

   Compile for '197.parser' ended at:Thu Dec 16 23:35:48 1999 (945405348)
   Elapsed compile for '197.parser': 00:02:45 (165)

   Build Complete

And that's it. The tools did most of the work; the user simply set PASS1_CFLAGS , PASS2_CFLAGS, and fdo_pre0 in the config file.

D. Help, I've got too many logs!

If you do a very large number of builds and runs, you may find that your result directory gets far too cluttered. If it does, you should feel free to issue commands such as these on Unix systems:

   cd $SPEC
   mv result result_old
   mkdir result

On NT, you could say:

   cd %SPEC%
   rename result result_old
   mkdir result

E. Finding the build directory

As described under "About Disk Usage" in runspec.html, the CPU2000 tools do the actual builds and runs in numbered run directories. The benchmark sources are never modified in the src directory.

To find the build directory, look for the word "build" in run/list. If more than one build directory is present, you can pipe the output to search for the specific extension that you want.

For example:

   F:\cpu2000> cd %SPEC%\benchspec\CINT2000\164.gzip\run

   F:\cpu2000\benchspec\CINT2000\164.gzip\run> findstr build list
   00000001 dir=F:/cpu2000/benchspec/CINT2000/164.gzip/run/00000001 
   ext=oct14a lock=0 type=build username=Administrator

   F:\cpu2000\benchspec\CINT2000\164.gzip\run> cd *01

F. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

G. For more information

For more information about how the run directories work, see the descriptions of specinvoke , specmake, and specdiff in utility.html.

VII. Troubleshooting

When something goes wrong with a build, here are some things to check:

1. Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward one screenful.

2. Did your desired switches get applied? Go to the build directory, and look at options*out.

3. Did the tools or your compilers report any errors? Look in the build directory at *err.

4. What happens if you try the build by hand? See the section on specmake in utility.html.

5. If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html

Copyright (C) 1999-2001 Standard Performance Evaluation Corporation All Rights Reserved