SPEC CPU2000 Utility Programs

Last updated: 29-Oct-2001 jh

(To check for possible updates to this document, please see http://www.spec.org/cpu2000/docs/ 





Contents






Introduction


   This document describes various useful utilities that ship with 
   SPEC CPU2000.  It is assumed that you have already read 
   runspec.html and config.html.







Note: links to SPEC CPU2000 documents on this web page
assume that you are reading the page from a directory that
also contains the other SPEC CPU2000 documents.  If by
some chance you are reading this web page from a location
where the links do not work, try accessing the referenced
documents at one of the following locations:

    

  • 
     www.spec.org/cpu2000/docs/

  • The $SPEC/docs/     directory on a Unix system where SPEC CPU2000 has been installed.

  •    The %spec%\docs.nt\ directory on a Windows/NT system where SPEC CPU2000 has been installed.

  •    The docs/           or
           docs.nt\        directory on your SPEC CPU2000 distribution cdrom.



















extract_config


If you have a rawfile, you can extract the config file that was used
with it.  To do so, use the extract_config utility.  


For example, starting with a freshly-installed
SPEC CPU2000 directory tree:

    % runspec --config=Tru64_Unix --size=test --iterations=1 \
      --tuning=peak --output_format=ps swim > /dev/null
    % cd result
    % ls
    CFP2000.001.ps   CFP2000.001.raw  log.001          log.lock  


Note that we have just the single result.  Let's make a temporary
directory and copy the rawfile there:

    % mkdir tmp
    % cd tmp
    % cp ../CFP2000.001.raw .
    % ls
    CFP2000.001.raw  


Now, attempt to extract the config file:

    % extract_config CFP2000.001.raw > tmp.cfg  


Does the extracted file match the original?

    % diff ../../config/Tru64_Unix.cfg ./tmp.cfg
    0a1,3
    > # Invocation command line:
    > # /cpu2000/kit96/bin/runspec --config=Tru64_Unix --size=test
    --iterations=1 --tuning=peak --output_format=ps swim
    >
    ############################################################################
    96,102d98
    <
    < __MD5__
    < 171.swim=peak=oct6a=default:
    < # Last updated Wed Dec 15 10:05:06 1999
    < optmd5=764e8d466d967f5dacae93415dbfbe71
    < exemd5=d7cccb73a236bbe7a548385ee4eff137
    <
    %  


Yes.  The only differences are:



NT Notes: on NT systems, you can invoke extract_config as:

    specperl %spec%\bin\extract_config rawfile   








extract_raw


If you have a pdf, html, or ps output file, you can extract the rawfile
that was used to generate it.  To do so, use extract_raw.


For example:

    % ls
    CFP2000.001.ps   CFP2000.001.raw  log.001          log.lock
    % mkdir tmp
    % cd tmp
    % cp ../*ps .
    % ls
    CFP2000.001.ps  


Note that at this point we have just the postscript output in the
current directory.  Now, attempt to extract the raw file:

    % extract_raw *ps
    % ls
    CFP2000.001.ps   CFP2000.001.raw  


Does the extracted file match the original?

    % diff ../CFP2000.001.raw ./CFP2000.001.raw
    %  


Yes.


NT Notes: on NT systems, you can invoke extract_raw as:

    specperl %spec%\bin\extract_raw file-to-extract-from   


Note that the file you extract from can be pdf, ps, or html, just
as on Unix systems, with one important exception: if you are using
a browser that attempts to add value to the html when you save it,
that browser may break the embedded rawfile.  For example, with 
Internet Explorer Version 5.00.3103.1000, if you are viewing a 
SPEC CPU2000 html file and you select File -> Save As, the default 
for "Save as type" is shown as 

     "Web Page, complete (*.htm,*.html)"  


Do not accept that default if you want to be able to retrieve the
rawfile later on.  If you take the default, your saved file will 
be incomplete.

To save the web page INCLUDING the embedded rawfile, you must change
"Save as type" to

     "Web Page, HTML only (*.htm,*.html)"  








go (Unix only)


When you source the shrc, as described in 
runspec.html, the
'go' shell alias is created.  Note: if you leave the shell where you
sourced shrc, for example by spawning csh, the alias is no longer 
available!  So if you want to use this alias, use a Bourne-compatible
shell.


For those of you who will have it, here are the various things it 
understands, as well as the places they'll take you. 
They're awfully handy.





If you say You'll end up in 


go             $SPEC            


go top         $SPEC            


 


go bin         $SPEC/bin  (where the tools live)            


 


go config      $SPEC/config            


 


go result      $SPEC/result            


go results     $SPEC/result            


 


go doc         $SPEC/docs            


go docs        $SPEC/docs            


 


go int         $SPEC/benchspec/CINT2000            


 


go fp          $SPEC/benchspec/CFP2000            


 


go src foo     foo's src directory, if it can be found.  For example:            


go src 197     $SPEC/benchspec/CINT2000/197.parser/src            


go src eon     $SPEC/benchspec/CINT2000/252.eon/src            


go src nada    .  (There's no benchmark named *nada*)            


 


go work foo     foo's run directory, if it can be found.  For example:            


go work 197     $SPEC/benchspec/CINT2000/197.parser/run            


go work eon     $SPEC/benchspec/CINT2000/252.eon/run            


go work nada    .  (There's no benchmark named *nada*)            


 


go data foo     foo's data directory, if it can be found.  For example:            


go data 197     $SPEC/benchspec/CINT2000/197.parser/data            


go data eon     $SPEC/benchspec/CINT2000/252.eon/data            


go data nada    .  (There's no benchmark named *nada*)            


 


go foo          foo's top level directory, if it can be found.  For example:            


go 197          $SPEC/benchspec/CINT2000/197.parser            


go eon          $SPEC/benchspec/CINT2000/252.eon            


go nada         .  (There's no benchmark named *nada*)            











printkids.pl


Some benchmark executables are invoked multiple times.  If you'd like to
know how much time was spent on each invocation, it may not be easy to
tell.  You could examine "speccmds.out", but that file is not easy to 
interpret:

    $ go work gcc
    $ cat list | fold -s
    00000001 dir=/cpu2000/kit92/benchspec/CINT2000/176.gcc/run/00000001
    ext=nov11c_ns lock=0 mach=default size=ref tune=base type=run
    username=john
    __END__
    $ cd *01
    $
    $ cat speccmds.out | fold -s
    running commands in speccmds.cmd 1 times
    runs started at 943371319, 399184000, Tue Nov 23 10:35:19 1999
    run 1 started at 943371319, 399184000, Tue Nov 23 10:35:19 1999
    child started: 0, 943371319, 400160000, 11128,
    '../00000001/cc1_base.nov11c_ns
    166.i -o 166.s'
    child finished: 0, 943371519, 814960000, 200, 414800000, 11128, 0
    child started: 0, 943371519, 815936000, 11135,
    '../00000001/cc1_base.nov11c_ns
    200.i -o 200.s'
    child finished: 0, 943371854, 339648000, 334, 523712000, 11135, 0
    child started: 0, 943371854, 339648000, 11154,
    '../00000001/cc1_base.nov11c_ns
    expr.i -o expr.s'
    child finished: 0, 943371892, 803248000, 38, 463600000, 11154, 0
    child started: 0, 943371892, 803248000, 11126,
    '../00000001/cc1_base.nov11c_ns
    integrate.i -o integrate.s'
    child finished: 0, 943371939, 571936000, 46, 768688000, 11126, 0
    child started: 0, 943371939, 571936000, 11146,
    '../00000001/cc1_base.nov11c_ns
    scilab.i -o scilab.s'
    child finished: 0, 943372136, 993568000, 197, 421632000, 11146, 0
    run 1 finished at: 943372136, 993568000, Tue Nov 23 10:48:56 1999
    run 1 elapsed time: 817, 594384000, 817.594384000
    runs finished at 943372136, 993568000, Tue Nov 23 10:48:56 1999
    runs elapsed time: 817, 594384000, 817.594384000  


Fortunately, SPEC has provided a tool to interpret the above for you:

    $ printkids.pl
      Seconds  Command
       200.41 cc1_base.nov11c_ns 166.i -o 166.s
       334.52 cc1_base.nov11c_ns 200.i -o 200.s
        38.46 cc1_base.nov11c_ns expr.i -o expr.s
        46.77 cc1_base.nov11c_ns integrate.i -o integrate.s
       197.42 cc1_base.nov11c_ns scilab.i -o scilab.s
    =========
       817.59 Total by adding children
       817.59 Total according to speccmds.out
    $  








relocate (Unix only)


If you rearrange mount points or rename your SPEC root directory, you 
will probably find confusing problems like the one in the example
that follows.  We used to have a working SPEC tree under /cpu2000/kit96,
but we're going to change that:

    $ cd /cpu2000
    $ mv kit96 larry
    $ cd larry
    $ . ./shrc
    $ runspec -h
    ksh: runspec: cannot execute  


The problem is that the first line of runspec points to a path that
no longer exists:

    $ head -1 bin/runspec
    #!/cpu2000/kit96/bin/specperl
    $  


Fortunately, SPEC has provided a tool to relocate the tools in the
new tree:

    $ bin/relocate
    Top of SPEC tree is '/cpu2000/larry'
    $ runspec -h | head -3
    runspec v2.00 - Copyright (C) 1999 Standard Performance Evaluation 
    Corporation
    Usage: bin/runspec [options]  


LIMITATIONS: there are some limitations to this tool.  Most notably,
it uses perl, so if the tree has been re-arranged in such a way that
perl has an identity crisis and can't find itself, you'll be in 
trouble.  Make sure that you have sourced shrc, as in the next 
example; if it still fails, you may need to reinstall the CPU2000 kit.

 
    $ mv larry carol
    $ cd carol
    $ bin/relocate
    Top of SPEC tree is '/cpu2000/carol'
    1062:/cpu2000/carol/bin/specperl: /sbin/loader: Fatal Error: cannot map
    libperl.so
    $ . ./shrc
    $ bin/relocate
    Top of SPEC tree is '/cpu2000/carol'
    $ runspec -h | head -2
    runspec v2.00 - Copyright (C) 1999 Standard Performance Evaluation
    Corporation
    Usage: bin/runspec [options]
    $  








rawformat 


"rawformat" is simply another name for 

   runspec --rawformat  


so please see runspec.html for basic 
information on using this command.
But a couple of additional points are worth mentioning:


    

  1. What do you do when Marketing decides to change the name from
       "SuperHero 4"
   to
       "SuperHero IV" 
   just before you submit the results to SPEC?
    

   
    SPEC does allow you to edit the rawfile to make corrections.  
   Just make sure that you only edit the fields located before
   the "fence":

         # =============== do not edit below this point ===================  
    

   WARNING: Make a backup copy of the rawfile first!  

   
    For example:

            copy  CFP2000.001.raw  new.raw
        (edit new.raw to make your changes)
        rawformat -o ps new.raw
        runspec v2.00 - Copyright (C) 1999 Standard Performance Evaluation
        Corporation
        Loading standard modules..............
        Loading runspec modules.............
        Identifying output formats...asc...config...html...pdf...ps...raw...
        Formatting new.raw
                format: PostScript -> new.ps  
    


  2.  About continuation fields in rawfiles
   
   
    Notice when editting fields that have more than one line that you
   need to construct the numbers with correct syntax.  The syntax in
   the rawfile is a bit more picky than in config files (compare the
   following vs. the config.html section on "Field scoping and
   continuation".  You cannot say something like this:
    

   
       WRONG:
     spec.cpu2000.sw_compiler: Intel C Compiler 4.5 for Windows 2000
     spec.cpu2000.sw_compiler1: Visual C++ 6.0 C (for libraries) 
    

   because the tools are not prepared at this point for a mixture of
   numbered and un-numbered fields.  If you need to have continued
   fields, say instead:
   
   
       RIGHT:
     spec.cpu2000.sw_compiler0: Intel C Compiler 4.5 for Windows 2000
     spec.cpu2000.sw_compiler1: Visual C++ 6.0 C (for libraries) 
    


  3.  How do you generate a 1-cpu rate result from a 1-cpu speed result?

   
    To generate a 1-cpu rate result from a speed run, copy the original
   rawfile to another location, and use rawformat to both generate the
   new rawfile and whatever other reports you want.  For example:

        $ grep SPECf CFP2000.015.asc
       SPECfp_base2000                        176
       SPECfp2000                                               --
    $ cp CFP2000.015.raw convertme
    $ rawformat --output_format asc,raw,ps --rate convertme
    runspec v2.00 - Copyright (C) 1999 Standard Performance Evaluation Corporation
    Loading standard modules..............
    Loading runspec modules.............
    Identifying output formats...asc...config...html...pdf...ps...raw...
    Formatting convertme
            format: ASCII -> convertme.asc
            format: raw -> convertme.raw
            format: PostScript -> convertme.ps
    $ grep SPECf convertme.asc
       SPECfp_rate_base2000             2.05
       SPECfp_rate2000                                    --  
    
    

    NT Notes: on NT systems, you may find
that --output_format will only accept one argument at a time.
So, first create the rawfile, by using --output_format raw, then use
the new rawfile to create the other reports.
    4. 









specdiff


In order to decide whether your system got the correct answer for a
benchmark, runspec writes a file called "compare.cmd" in the run
directory, and then calls specinvoke, which calls specdiff.  (The
specinvoke utility is described 
later in this document.)


Suppose that your results are flagged as incorrect, but you'd like 
to have more detail.  You could ask for more than 10 lines of
detail by changing the config file's default for 
difflines from 10 to 
something else, and re-running the experiment.
But, you might save a lot of time and avoid the re-run of
the experiment, by calling specdiff directly, with a different
number of lines for its switch -l.


For example, 
a file chock-full of wrong answers for ammp has been created by
hand.  First, let's see how specdiff would normally be run (line 
wraps added for readability):

    $ specinvoke -c 1 -f compare.cmd -n
    /cpu2000/kit96/bin/specdiff -m -l 10  -a 0.0001  -r 0.003
    /cpu2000/kit96/benchspec/CFP2000/188.ammp/data/test/output/ammp.out 
    ammp.out > ammp.out.cmp  


The above says that normally we test for correct answers by comparing
data/test/output/ammp.out to ammp.out in the current directory.
Normally, 10 lines of detail are printed.  Let's change that to 15
lines, and compare versus our intentionally wrong file instead:

    $ /cpu2000/kit96/bin/specdiff -m -l 15 -a 0.0001  -r 0.003 \
    /cpu2000/kit96/benchspec/CFP2000/188.ammp/data/test/output/ammp.out \
    ammp.wrong_output
    0002:   setf mxdq 0.75;
            setf mxdq 01.75;
                          ^
    0003:    setf mmbox 10.;
             setf mmbox 101.;
                           ^
    0006:   setf bbox 35.;
            setf bbox 351.;
                         ^
    0009:   0.000000 unknown potential type
            01.000000 unknown potential type
                    ^
    0010:   1181.693079 bond energy
            11811.693079 bond energy
                       ^
    0011:   2056.767980 angle energy
            20561.767980 angle energy
                       ^
    0012:   61.501074 hybrid energy
            611.501074 hybrid energy
                     ^
    0013:   533.960821 torsion energy
            5331.960821 torsion energy
                      ^
    0014:   -15410.005122 non-bonded energy
            -154101.005122 non-bonded energy
                         ^
    0015:   -11576.082168 total potential energy
            -115761.082168 total potential energy
                         ^
    0016:   8584.512662 total kinetic energy
            85841.512662 total kinetic energy
                       ^
    0017:   -2991.569506 total energy
            -29911.569506 total energy
                        ^
    0018:   20160.594829 total action
            201601.594829 total action
                        ^
    0020:   tpac numstp .00001 temp;
            tpac numstp 1.00001 temp;
                              ^
    0022:   0.000000 unknown potential type
            01.000000 unknown potential type
                    ^
    0023:   1184.181101 bond energy
            11841.181101 bond energy
                       ^
    $  


Here are the switches supported by specdiff:

    $ specdiff -h
    Usage: /cpu2000/kit96/bin/specdiff [-l #] [-q] file1 [file2]
       -l     # of lines of differences to print
       -q     don't print lines just set return code
       -a     absolute tolerance (for floating point compares)
       -r     relative tolerance (for floating point compares)
       -t     set calctol
       -s     set skiptol
       -o     allow off-by-one errors
       -O     *don't* allow off-by-one errors
       -m     write file2.mis with miscompares
       -c     collapse whitespace (doesn't do what you think it does)
       -C     *don't* collapse whitespace
       -d     Compare against file(s) in data directory
      --os    Set the operating system type (you don't need to do this)
       -v     Set the level of noisiness for the output
       -h     Print this message
    $  








specinvoke


In order to run a benchmark, runspec writes a series of commands 
in the run directory, in the file "speccmds.cmd".  These commands
are then interpreted by specinvoke.

When something goes wrong, it is often useful to try the commands by
hand.  Ask specinvoke what it did, by using the "-n" switch:

    $ go work swim
    $ cd 00000002
    $ ls
    SWIM7            compare.out      swim.err         swim_peak.oct6a
    SWIM7.cmp        speccmds.cmd     swim.in
    compare.cmd      speccmds.err     swim.out
    compare.err      speccmds.out     swim.out.cmp
    $ specinvoke -n
    ../00000002/swim_peak.oct6a < swim.in > swim.out 2> swim.err
    $  


In the above example, we see how swim is invoked.  If you wanted
to enter your own variations on the above command by hand, you 
could do so.

Here are the switches supported by specinvoke:

    $ specinvoke -h
    Usage: specinvoke [options] [instructionfile]
           -i #      iterations
           -c #      concurrent processes (overrides -u in command file)
           -E        return non-zero exit code if child does
           -s shell  shell to invoke
           -f file   instruction file
           -o file   output file
           -e file   error file
           -d dir    change to dir first
           -n[n]     print a 'dry_run' of commands
           -A        return 0 error code
	   -r        don't do command I/O redirection ($command already has it)
           -h        this message  









specmake


To build a benchmark, runspec uses specmake, which is simply gnu make
under a unique name to avoid possible conflicts with other versions of 
make on your system.

If a benchmark build fails, the most massively useful thing you can do
is to ask specmake what it did, by using the "-n" switch for dry run.

For example, let's create a config file that is unlikely to work:

    $ cat >tmp.cfg
    ext = silly
    FC  = f66
    $  


and try it out:

    $ runspec --config=tmp --tuning=base --action=build swim
    runspec v2.00 - Copyright (C) 1999 Standard Performance Evaluation
    Corporation
    Loading standard modules..............
    Loading runspec modules.............
    Identifying output formats...asc...config...html...pdf...ps...raw...
    We will use: 171.swim
    Compiling Binaries
      Building 171.swim ref base silly default
    Error with make 'specmake  build > make.out 2> make.err': check file
    '/cpu2000/kit96/benchspec/CFP2000/171.swim/run/00000004/make.err'
      Error with make!
    *** Error building 171.swim
    If you wish to ignore this error, please use '-I' or ignore errors.
    $  


Sure enough, it failed.  Let's find the appropriate run directory 
where our silly build was attempted:

    $ go run swim
    $ grep build list | grep silly | cut -b 1-9
    00000004
    $ cd 00000004   


And ask specmake just exactly what it generated:

    $ specmake -n
    f66 -c -o swim.o                swim.f
    f66         swim.o     -o swim  


OK, let's enter the command by hand:

    $ f66 -c -o swim.o                swim.f
    ksh: f66:  not found
    $  



A great convenience of the CPU2000 suite is the fact that you can now 
proceed to play with all the files in this run directory, to prototype 
different changes and try out ideas, without fear that you will break 
anything.  As long as you confine your edits to (in this example) 
$SPEC/benchspec/CFP2000/171.swim/run/00000004/, you are in a private 
sandbox and will not harm the original sources.


Here are the switches supported by specmake:

$ specmake -h
Usage: specmake [options] [target] ...
Options:
  -b, -m                      Ignored for compatibility.
  -C DIRECTORY, --directory=DIRECTORY
                              Change to DIRECTORY before doing anything.
  -d, --debug                 Print lots of debugging information.
  -e, --environment-overrides
                              Environment variables override makefiles.
  -f FILE, --file=FILE, --makefile=FILE
                              Read FILE as a makefile.
  -h, --help                  Print this message and exit.
  -i, --ignore-errors         Ignore errors from commands.
  -I DIRECTORY, --include-dir=DIRECTORY
                              Search DIRECTORY for included makefiles.
  -j [N], --jobs[=N]          Allow N jobs at once; infinite jobs with no arg.
  -k, --keep-going            Keep going when some targets can't be made.
  -l [N], --load-average[=N], --max-load[=N]
                              Don't start multiple jobs unless load is below N.
  -n, --just-print, --dry-run, --recon
                              Don't actually run any commands; just print them.
  -o FILE, --old-file=FILE, --assume-old=FILE
                              Consider FILE to be very old and don't remake it.
  -p, --print-data-base       Print make's internal database.
  -q, --question              Run no commands; exit status says if up to date.
  -r, --no-builtin-rules      Disable the built-in implicit rules.
  -s, --silent, --quiet       Don't echo commands.
  -S, --no-keep-going, --stop
                              Turns off -k.
  -t, --touch                 Touch targets instead of remaking them.
  -v, --version               Print the version number of make and exit.
  -w, --print-directory       Print the current directory.
  --no-print-directory        Turn off -w, even if it was turned on implicitly.
  -W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE
                              Consider FILE to be infinitely new.
  --warn-undefined-variables  Warn when an undefined variable is referenced.  






Copyright (C) 1999-2000 Standard Performance Evaluation Corporation

All Rights Reserved