SPEChpc™ 2021 Utilities

Contents

Introduction

This document describes various useful utilities that ship with SPEChpc 2021.

configpp

configpp preprocesses a config file.
The preprocessor is automatically run every time that you use runhpc.
Running it separately is a good way to check whether your config file macros do what you hope they do.

configpp --config=file [--define symbol[=value]]... [--output=file]
configpp -c file       [-S symbol=[value]]...       [-o file]

Tips and Traps

For much more information on the preprocessor, see its chapter chapter in SPEChpc 2021 Config Files.

convert_to_development

convert_to_development removes the SPEC safety checks which normally would verify the integrity of benchmark code, data, and tools.
Before using this utility, you should carefully consider the Tips and Traps below.

$ convert_to_development

Development tree conversion: /tmp/hpc2021

After this conversion, the benchmark tree named above will no longer be
usable for reportable runs.  In return, all of the safety features that
ensure that sources, data, and tools are not changed are removed.

This conversion is not reversable.  In order to regain the ability to do
reportable runs, it will be necessary to do a complete new reinstallation in
another location; re-running install.sh or install.bat will not be sufficient.

Do you still want to continue with this conversion?
yes
Successfully performed development tree conversion.
$ 

An academic or research user might consider using convert_to_development while investigating performance effects of alternate source code and data sets.

Tips and Traps

dumpsrcalt

dumpsrcalt displays the contents of a src.alt (a compressed tar file created by makesrcalt), without necessarily having to apply it.

dumpsrcalt [--summarize] src.alt 

To find out what is in a src.alt, it may be easiest to just apply it.
Or, you can use dumpsrcalt, which displays the differences in a form that will be familiar to users of patch utilities. To find out more about the utility, please see its --help.

extract_config

extract_config reads a rawfile, decodes the config file that produced it, and writes the config file to a file or to standard output.

extract_config rawfile

The extracted config file will have these differences vs. the original:

• Added: a comment at the top to say where it came from
• Added: data from sysinfo
• Removed: protected comments
• Removed: the HASH section

An extracted config file will exactly match a config file as printed by rawformat --output_format=config with one exception: if you have edited the raw file, it will not include your edits.

Example: first, a config file is extracted:

$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_config $SPEC/result/hpc2021_sml.009.small.ref.rsf
Wrote "hpc2021_sml.009.small.ref.cfg"
$ mv hpc2021_sml.009.small.ref.cfg from_extract_config
   

The extracted config file was moved out of the way, so it won't be over-written when we run rawformat:

$ specperl $SPEC/bin/formatter/rawformat --output_format=config,nocheck $SPEC/result/hpc2021_sml.009.small.ref.rsf | grep format:
    format: cfg -> hpc2021_sml.009.small.ref.cfg, hpc2021_sml.009.small.ref.orig.cfg
   

If you edit a rawfile, the .orig.cfg shows what the config file looked like prior to editing. It matches the extracted config file (with the exception of having one extra line).

$ diff hpc2021_sml.009.small.ref.cfg from_extract_config 
454a455
> 
$  

extract_flags

extract_flags reads a rawfile, decodes any flags files that are stored within it, and writes them to one of several destinations.

extract_flags rawfile

A flags file has a concept of a suggested filename. When writing an output file, that is the first preference.
A second choice would be to match the name of the input file.
A final choice would be to specify an arbitrary new name.
Each of these is shown in the examples below (use --help for a description of the switches).

$ extract_flags hpc2021_sml.009.small.ref.rsf 
Reading "hpc2021_sml.009.small.ref.rsf"
Wrote "gcc.xml"
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_flags $SPEC/result/hpc2021_sml.009.small.ref.rsf
Reading "result/hpc2021_sml.009.small.ref.rsf"
Wrote "result/nv2021_flags.xml"
$ cat $SPEC/result/hpc2021_sml.009.small.ref.rsf | specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_flags --nofilename --basename=rutabaga
Reading from stdin
Wrote "rutabaga.xml" 
$

extract_raw

extract_raw reads an HTML, PDF, or PostScript report and extracts the rawfile that generated it.

extract_raw <report>

The raw file is the most important of the outputs.
If you have it, you can re-generate any of the others using rawformat.

$ ls
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_raw $SPEC/result/hpc2021_sml.009.small.ref.pdf
result/hpc2021_sml.009.small.ref.pdf: Wrote "hpc2021_sml.009.small.ref.rsf"
$ specperl $SPEC/bin/formatter/rawformat --output_format=ps,html,txt,csv,config,nocheck hpc2021_sml.009.small.ref.rsf > /dev/null
$ ls result/hpc2021_sml.009.small*
result/hpc2021_sml.009.small.ref.cfg         result/hpc2021_sml.009.small.ref.orig.cfg  result/hpc2021_sml.009.small.ref.txt
result/hpc2021_sml.009.small.ref.flags.html  result/hpc2021_sml.009.small.ref.pdf       result/hpc2021_sml.009.small.test.rsf
result/hpc2021_sml.009.small.ref.html        result/hpc2021_sml.009.small.ref.rsf       result/hpc2021_sml.009.small.test.txt
$  

flag_dump

flag_dump reads an XML flag description and creates a human-readable HTML version from it.

flag_dump <XML flags file>...
         

If you are writing a flags file, you should both:

1. Verify XML correctness with specrxp.

2. Verify HTML correctness of its included HTML components:

   • Use flag_dump
   • Send its output to an HTML validator, such as http://validator.w3.org/.

Example:

$ ls result/*.xml
result/nv2021_flags.xml
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/flag_dump  $SPEC/result/nv2021_flags.xml
Processing result/nv2021_flags.xml...wrote nv2021_flags.html
$  

go (Unix only)

go provides a handy alias to navigate a SPEChpc 2021 directory tree.
If you just say go (without any argument) your directory is set to $SPEC.
Other destinations are described in the table below.

go destination  [subdir]...  

top              $SPEC
bin              $SPEC/bin
config           $SPEC/config
Docs|docs        $SPEC/Docs
result           $SPEC/result

nnn.benchmark    $SPEC/benchspec/CPU/nnn.benchmark
nnn              Same, if nnn matches a benchmark
substring        Same, if substring matches the start of a benchmark name;
{build|exe|run}  If you are already under some  nnn.benchmark, goes to
                 nnn.benchmark/{build|exe|run} 

Notes

• If you start a sub-shell, the alias probably will not work until you once again source shrc or cshrc.
• The alias is unlikely to work in scripts. Check your shell documentation.
• go does not know directories that are created by expid.
• go does not know about output_root directories, but ogo does!

Examples

$ go
/spec/hpc2021/
$ go result
/spec/hpc2021/result
$ go 605 exe
/spec/hpc2021/benchspec/HPC/605.lbm_s/exe
   

If you abbreviate a benchmark name, the first matching benchmark is used.
Below, it would be easier to just type go 635.

$ go weather exe
/spec/hpc2021/benchspec/HPC/635.weather_s/exe
$ go tealeaf_s exe
/spec/hpc2021/benchspec/HPC/618.tealeaf_s/exe
$ go build
/spec/hpc2021/benchspec/HPC/618.tealeaf_s/build
   

You can string subdirectories together.

$ go bin lib Math BigInt
/spec/hpc2021/bin/lib/Math/BigInt
$   

index (UNSUPPORTED)

index prepares an HTML index for a set of results.

      $SPEC/bin/scripts.misc/index

As you are preparing sets of results, you might find it handy to use scripts.misc/index. The utility is officially unsupported because its primary intent is for use at the SPEC website; features may therefore be added, subtracted, or adjusted for convenience of the SPEC webmasters, rather than for user convenience. It might work for you; or you might use it as a basis for writing your own utility. You are welcome to try it out, but please do not submit support requests against it.

$ cd $GO/result
$ mkdir prep_for_pub
$ cp $SPEC/result/hpc2021_sml.009.small.ref.rsf prep_for_pub/TurboBlaster9000.small.mpi.ref
$ cp $SPEC/result/hpc2021_sml.010.small.ref.rsf prep_for_pub/TurboBlaster9000.small.omp.ref
$ cp $SPEC/result/hpc2021_sml.011.small.ref.rsf prep_for_pub/TurboBlaster9000.small.acc.ref
$ cp $SPEC/result/hpc2021_sml.014.small.ref.rsf prep_for_pub/TurboBlaster9000.small.tgt.ref
$ cd prep_for_pub/
$ vi *rsf 

$ rawformat --flags $SPEC/config/flags/gcc.xml \
  -output_format=all *rsf > rawformat.log             
$ $SPEC/bin/scripts.misc/index
$ ls index.html
index.html
$ 

makesrcalt

makescralt packages alternative source into a compressed tar file, known as a src.alt.

mkdir         nnn.benchmark/src/src.alt/name
cp sources... nnn.benchmark/src/src.alt/name
cp README     nnn.benchmark/src/src.alt/name
makesrcalt    nnn.benchmark name

SPEC will post approved SPEChpc 2021 src.alts at www.spec.org/hpc2021/src.alt.
Users apply them via the config file src.alt option.

src.alt Development

makesrcalt Example

In this example, a comment is added to a source file:

$ pwd
/hpc2021/benchspec/HPC/605.lbm_s/src/src.alt/addAcomment
$ ls
README         lbm.c 
$ diff -u ../../lbm.c lbm.c 
80a81,83
> // Test src.alt example
> // This is only a test
>
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/makesrcalt 605.lbm_s addAcomment
Making 'addAcomment' src.alt for 605.lbm_s

Making differences and computing hashes...
Writing control file...
Gathering files...
Wrote /hpc2021/605.lbm_s.addAcomment.hpc2021.kit0.2.tar.xz

$ (Output subsetted for brevity.) 

ogo (Unix only)

ogo provides a handy alias that navigates *both*

• $SPEC - a SPEChpc 2021 directory tree
• $GO - a related tree created by output_root

ogo automatically picks a tree and a directory depending on whether you want to move to a read-only directory (e.g. lbm src) or an output directory (e.g. lbm build).

ogo destination  [subdir]...  

Notes

• If you start a sub-shell, the alias probably will not work until you once again source shrc or cshrc.
• The alias is unlikely to work in scripts. Check your shell documentation.
• ogo does not know directories that are created by expid.
• New with SPEChpc 2021: %{ENV_GO} can be usefully combined with ogo

Set GO once, at login.
Config files can pick it up from an ENV macro

$ grep GO ~/.bashrc
export GO=/tmp/outputs
$ cat autoroot.cfg
output_root          = %{ENV_GO}  # pick up automatically from environment
iterations           = 1
label                = xyzzy
runlist              = 605.lbm_s
size                 = test
default:
   CC                = mpicc
   CC_VERSION_OPTION = -v
$ cat autoroot.sh 
runhpc --config=autoroot | grep txt
$ ./autoroot.sh 
    format: Text -> /tmp/outputs/result/hpc2021_sml.001.small.test.txt

ogo handily moves around in the directories from the above command.
It confirms your destination underneath $GO.

$ ogo
Using value in GO for output_root: /tmp/outputs
/tmp/outputs
$ ogo mcf run
Using value in GO for output_root: /tmp/outputs
/tmp/outputs/benchspec/HPC/605.lbm_s/run
$ ogo exe
Using value in GO for output_root: /tmp/outputs
/tmp/outputs/benchspec/HPC/605.lbm_s/exe
$ ls
lbm_s_base.xyzzy

If you get tired of the two-line message, you can shrink it by setting OGO_NO_WARN.

$ OGO_NO_WARN="quiet please" 
$ ogo lbm run
/tmp/outputs/benchspec/HPC/605.lbm_s/run

ogo uses the tree rooted at $SPEC if the destination is read-only (lbm src).

$ ogo lbm src
/spec/hpc2021/benchspec/HPC/605.lbm_s/src
$  

port_progress

port_progress analyzes log files and summarizes errors. It was primarily used during the benchmark porting phase for SPEChpc 2021 (thus its name), but it can still come in handy, for example when trying the benchmarks with a new compiler.

port_progress [-b benchmarks] [-c config] [-q] [--rate] [--skipnt] [--speed] [--table] [-a|logs...] 

port_progress quickly summarizes the completion status for a series of runs. The report is organized into columns laid out by config file, tuning (base or peak), and workload size (Te for test, Tr for train, Ref for ref). Each benchmark and workload is summarized using codes such as:

S  - Success
CE - Compile error
RE - Run time error (did not finish; crashed or core dumped)
VE - Verification Error (it finished, but got the wrong answer)

In the example below, the tester has been trying several compiler versions (X, Y, and Z) using several config files. A report is requested only for the troublesome benchmarks - all of which worked with the most recent compiler.

$ specperl $SPEC/bin/tools/port_progress -q -a --skipnt --table -b 'lbm|tealeaf|pot3d|weather' result/*.log
Looking for benchmarks that match 'lbm|tealeaf|pot3d|weather'
SPEC hpc2021 v0.2.7
  Wed Oct 21 13:24:55 2020
Tester: tony   System: turbo5

                ------  ------  ------  ------  
                acc     mpi     omp     tgt  
                ------  ------  ------  ------  
                Base    Base    Base    Base    
                ------  ------  ------  ------  
                Te Ref  Te Ref  Te Ref  Te Ref  
                -- ---  -- ---  -- ---  -- ---  
605.lbm_s       S  S    S  S    S  S    S  S    
618.tealeaf_s   S  S    S  S    S  S    S  S    
628.pot3d_s     S  S    S  S    S  S    S  S    
635.weather_s   S  S    S  S    S  S    S  S    
   

printkids.pl

printkids.pl prints a table showing the execution times for each of a benchmark's subcomponents. Before using it, cd to a run directory.

printkids.pl [--columns=N] [--truncate] [--wrap] 
Windows: %SPEC%\bin\specperl %SPEC%\bin\tools\printkids.pl &etc

Some SPEC benchmark executables are invoked multiple times. If you'd like to know how much time was spent on each invocation, you can examine the log file, or you can go to a run directory and use printkids.pl. In the example below from SPEC CPU2017, the ref run for 525.x264_r has three parts, and it is apparent that the first part is the shortest.

$ ogo 525 run
Using value in GO for output_root: /Users/somebody/spec/cpu2017/out.rc5
/Users/somebody/spec/cpu2017/out.rc5/benchspec/CPU/525.x264_r/run
$ cd run_base_refrate_mar10a-m64.0000/
$ printkids.pl --wrap --columns=90
  Seconds  Command
    46.58 x264_r_base.mar10a-m64 --pass 1 --stats x264_stats.log --bitrate 1000 --frames
                                 1000 -o BuckBunny_New.264 BuckBunny.yuv 1280x720
   159.59 x264_r_base.mar10a-m64 --pass 2 --stats x264_stats.log --bitrate 1000 --dumpyuv
                                 200 --frames 1000 -o BuckBunny_New.264 BuckBunny.yuv
                                 1280x720
   162.37 x264_r_base.mar10a-m64 --seek 500 --dumpyuv 200 --frames 1250 -o
                                 BuckBunny_New.264 BuckBunny.yuv 1280x720
=========
   368.54 Total by adding children      
   368.54 Total according to speccmds.out
$ 

printpath.pl

printpath.pl is a script that can be used on Windows to print the path in a format that is easier to read.
You can send the output to a file, which can then be easily edited.

NOTE: printpath.pl is not supported with SPEChpc 2021

specperl %SPEC%\bin\windows\printpath.pl

In the example below, printpath.pl makes it easier to notice that cygwin (a Windows/Unix compatibility product) is in the path. SPEC recommends that your %PATH% should not include Windows/Unix compatibility products, because these have sometimes contributed to hard-to-diagnose errors (link goes to a CPU2006 FAQ).

C:\cpu2017\rc5>echo %PATH%
C:\cpu2017\rc5\bin\windows;C:\cpu2017\rc5\bin;C:\ProgramData\Oracle\Java\javapat
h;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\Wi
ndowsPowerShell\v1.0\;c:\cygwin\bin;c:\unison

C:\cpu2017\rc5>specperl %SPEC%\bin\windows\printpath.pl
PATH=^
C:\cpu2017\rc5\bin\windows;^
C:\cpu2017\rc5\bin;^
C:\ProgramData\Oracle\Java\javapath;^
C:\windows\system32;^
C:\windows;^
C:\windows\System32\Wbem;^
C:\windows\System32\WindowsPowerShell\v1.0\;^
c:\cygwin\bin;^
c:\unison
   

rawformat

rawformat is the report writer for SPEChpc 2021. It creates HTML, PDF, and many other forms of output.
rawformat is automatically invoked by runhpc at the end of a performance test.
It can be invoked directly, by typing rawformat, or equivalently by typing runhpc --rawformat.
(The name rawformat indicates that its input is a raw data file generated by an experiment, which it transforms into various formats.)

rawformat [options] rawfiles...

You might use rawformat many times just before completing a project, as you go through cycles of the form
rawformat-review-edit-rawformat-review-edit-rawformat...
until finally you get to ...publish!

Editing rawfiles (ONLY above the fence!)

# =============== do not edit below this point =================== 

$ grep hw_model $SPEC/result/hpc2021_sml.015.small.ref.rsf 
spec.hpc2021.interconnect_fs_hw_model: BI 100 Series
spec.hpc2021.node_compute_hw_model: Turblaster 9000
spec.hpc2021.node_fileserver_hw_model: BG650
$ mkdir prep_for_publication
$ cp $SPEC/result/hpc2021_sml.015.small.ref.rsf prep_for_publication/tb9000.hpc2021.sml.rsf
$ cd prep_for_publication/

$ vi tb9000.hpc2021.sml.rsf 
$  diff $SPEC/result/hpc2021_sml.015.small.ref.rsf tb9000.hpc2021.sml.rsf
   36c36
< spec.hpc2021.node_compute_hw_model: Turblaster 9000
---
> spec.hpc2021.node_compute_hw_model: Turbo Blaster 9000

$ specperl $SPEC/bin/formatter/rawformat --output_format txt,html,csv tb9000.hpc2021.sml.rsf

Continued fields

When editing fields that have more than one line, you cannot mix numbered and un-numbered versions of a single field. Instead, you must use a number on both of them:

spec.hpc2021.sw_compiler: C/C++: Version 17.0.0.098 of Intel C/C++
spec.hpc2021.sw_compiler1: Compiler for Linux

spec.hpc2021.sw_compiler000: C/C++: Version 17.0.0.098 of Intel C/C++
spec.hpc2021.sw_compiler001: Compiler for Linux

Test_date: field allowed only in rawfiles

For SPEChpc 2021, one field is ignored in config files, but may be set in a raw file:

specdiff

specdiff determines whether benchmark outputs match the SPEC-supplied expected outputs.
Benchmark developers define the validation criteria by specifying acceptable tolerances.

specdiff [--abstol n] [--calctol] [--obiwan] [--reltol n] [--skiptol n] file1  file2

Tolerances are written by runhpc to compare.cmd, which specinvoke reads when it calls specdiff.
If you are debugging a problem, you can look at how the validation is done and you can run specdiff by hand.

Example: Go to a run directory and find out what specdiff command would be generated using specinvoke's dry-run switch, -n.

$ go weather run
$ cd run_base_test_omp.0000 
$ specinvoke -n compare.cmd | grep specdiff
$SPEC/bin/specperl $SPEC/bin/harness/specdiff -m -l 10 --abstol 4e-07 --reltol 0.006 $SPEC/benchspec/HPC/535.weather_t/data/ref/output/output6.ref.txt output6.ref.txt 0<&- > output6.ref.txt.cmp
$  $SPEC/bin/specperl $SPEC/bin/harness/specdiff -m -l 10 --abstol 4e-07 --reltol 0.006 --calctol $SPEC/benchspec/HPC/535.weather_t/data/ref/output/output6.ref.txt output6.ref.txt
Calculated tolerances:
Maximum abstol: 1.00002e-10
Minimum abstol: 9.99947e-11
# of abstol errors: 0
Maximum reltol: 1.20429e-08
Minimum reltol: 2.38284e-12
# of reltol errors: 0
# of obiwan errors: 0
# of skiptol errors: 0
specdiff run completed
$

Caution to benchmark developers: The --calctol switch answers two separate questions, one at time: (1) Given all the other tolerances listed, what relative tolerance R would be needed in order for these files to be considered a match? and (2) Given all the other tolerances listed, what absolute tolerance A would be needed? Notice that:
Any particular difference D that is just barely covered by R may be much smaller than A.
Any particular difference D that is just barely covered by A may be much smaller than R.
Therefore, a benchmark developer who is considering a tolerance change should never blindly apply the results of --calctol. Consider a much smaller change, and recalculate.

specinvoke

specinvoke runs and times commands, after runhpc sets them up.
It can display commands needed to set up the environment and to change directories.

specinvoke [options] [commandfile]

Environment variables: It is important to understand that specinvoke does not set environment variables; but you can cause them to be set by capturing the dry run output and executing it. The output of specinvoke -nn includes commands to:

• Set the current directory.
• Add environment variables that were in the environment when runhpc set up the directories.
• For OpenMP or OpenACC, set OMP_NUM_THREADS and ACC_NUM_CORES to the i --threads command line value or to the config file threads= value.
• Remove extraneous environment variables that are present now but were not present when runhpc was run.

• A final line is also included which requires notice:
  specinvoke exit: rc=0

  • This must be removed if you execute the dry run commands.
  • If you do not, then specinvoke will try to re-run itself, looking for a file named exit: that it is unlikely to find.
  • See Example 3 below.

The specinvoke utility by default looks for speccmds.cmd, which is how benchmarks are run. However, that is not its only use. The runhpc program also uses specinvoke for setting up certain benchmarks, via inputgen.cmd; and to drive comparisons, as described in the section on specdiff. You might want to use it to run benchmarks "by hand", or to customize the runs during a performance analysis exercise.

specmake

specmake is GNU make under another name (to avoid any possible conflicts with other versions of make on your system). It builds benchmark executables, driving compilers/linkers/other build components as needed. Typically, you do not use specmake directly; instead, you specify make variables in your config file. However, if something goes wrong, a useful debug strategy is to cd to the build directory, and try specmake directly until you figure out what the problem is.

specmake [options] [target]...

When something goes wrong with a build, you can use the build directory as a "sandbox", trying out different therapies. Examples:

cd nnn.benchmark/build/build_where_something_broke.0000
specmake -n | wc -l

cd nnn.benchmark/build/build_where_something_broke.0000
specmake --keep-going 

cd nnn.benchmark/build/build_where_something_broke.0000
specmake clean
specmake CXXOPTIMIZE=-O2 

cd nnn.benchmark/build/build_where_something_broke.0000
edit Makefile.spec # try This
specmake           # .......... still fails? 
edit Makefile.spec # try That
specmake           #  .......... still fails?
edit Makefile.spec # try Other
   ...&etc

specmake preprocess creates both Fortran intermediate files (name.fppized.f90) and C/C++ intermediate files (name.i).
These have very differing existence requirements, usefulness, and relevance:

Fortran preprocessed files are always created. Fortran preprocessing for SPEChpc 2021 benchmarks is never done by the compilers; it is done by specpp. Your Fortran compiler is not called until after name.fppized.f90 has been created. A reason you might say specmake preprocess is because you want to create all of them at once. Because your Fortran compiler does not see the original files, the executable always corresponds to the preprocessed source, and any line numbers reported by debugger messages may differ greatly from original source line numbering.

C/C++ preprocessed files are rarely created. C/C++ preprocessing is done by compilers, which may vary in how they do it. Typically, C/C++ compilers do not bother to save preprocessed output to intermediate files (name.i) unless requested to do so. A reason you might want to create them is in order to verify whether macros are behaving as expected; however, in some cases, the compiler might not promise that it is even capable of creating a name.i file that corresponds exactly to the object. Line numbers reported by debuggers normally match the line numbering of the original source, not the numbering of name.i files.

specperl

specperl is simply Perl, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path. Your system probably already has Perl -- but it might be older or newer, or might not include the same CPAN additions.
You can compare your system Perl to specperl by comparing the output of "perl -V" vs. "specperl -V".

Interactive use: You can use specperl interactively if you have executed one of the three path-setting scripts (see list in Quick Start).
You can get help using specperldoc.
Examples:

specperl -v                           # print basic version info
specperl -V                           # print version details
specperl -e "print hex '1C000800'"    # One liner to convert a number
specperldoc perlre                    # Find out about regular expressions 
                                      # (useful for flag description files)
specperldoc -f hex                    # Lookup a function

Warning: specperl might not be the best choice for your own custom scripts. If you have a system Perl, it may be better adapted to the particular characteristics (or idiosyncracies) of your particular operating system and hardware. Caveat programmer, use at own risk.

How do the tools use specperl? When using Perl, a key requirement is to set paths correctly. For tools such as

the paths are set by tiny, similarly-named scripts in either $SPEC/bin
These set the path, then exec or call the real scripts, using something similar to

specperl -IdirectoryThis IdirectoryThat realScript

specpp

specpp is the Fortran preprocessor.

specpp [-Dmacro] [-Idir] [-m c-comment.pm] [-w] 

Preprocessing is not a part of standard Fortran. Therefore, SPEC supplies a copy of Darren Miller's freely available preprocessor utility filepp, with minor modifications, as specpp. Using the same preprocessor on all platforms ensures that the preprocessing is done consistently.

When building benchmarks, your Fortran compiler is not invoked until after specpp has completed its work. Therefore, config.html points out:

• FPORTABILITY     is not the right place to set Fortran preprocessor variables
• FPPPORTABILITY   is     the right place.

For example, if a config file sets

FPORTABILITY   = --files=bigendian
FPPPORTABILITY = -DSPEC_ACCEL_AWARE_MPI
   

and if a benchmark uses mumble.F90, then specpp will create mumble.fppized.f90 prior to the invocation of the Fortran compiler. The preprocessor will see the flag -DSPEC_ACCEL_AWARE_MPI. The compiler will see the flag --files=bigendian and the compiler not see the flag -DSPEC_ACCEL_AWARE_MPI.

Usually, you won't need to know any more about specpp, because it is automatically invoked. If you are a benchmark developer, or if you are simply curious, the manpage is available.

Don't confuse specpp with configpp.

specrxp

specrxp validates the XML in a flags file.
It is simply Richard Tobin's RXP under a unique name to ensure no possible conflicts with other versions of rxp on your system.

specrxp -V -s <flags file>

If you write a flags file, you should validate that you have written well-formed XML, using specrxp, typically with -V to request validation and -s to request silence (if there are no errors). Many more options can be found in the manpage.

q. Why does specrxp seem to hang?

a. You are behind a firewall. The following line, which must be present at the top of your flags file, needs internet access:

   SYSTEM "http://www.spec.org/dtd/cpuflags2.dtd"
   

To resolve the problem, you can:

1. Copy the flags file to a system that does have access.
2. Or define a proxy (e.g. export http_proxy=http://www-proxy.mycompany.com:80/)
3. Or temporarily change the line to point at a local file. Change it back afterwards!

Reminder: after validating the XML, dump and then validate your HTML. Both steps are needed.

specmd5sum, specsha256sum, specsha512sum

specmd5sum, specsha256sum, and specsha512sum calculate and verify checksums. They are based on
md5sum, sha256sum, and sha512sum from GNU coreutils.

spectar

spectar is simply GNU tar, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path. GNU tar, unlike some other archiving programs, correctly handles very long path names.

specxz

specxz is xz from XZ Utils, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path.

Commonly-used switches work analogously to those in other compression utilities, such as -d to decompress. For details, say specxz --help

Warning: this idiom, commonly used on Unix systems, has been observed to fail on some tested Windows systems:

specxz -dc mumble.tar.xz | spectar -xf -  <-- may fail on Windows
   

If you see failures, break it into two commands.

specxz -d mumble.tar.xz                   <-- an alternative
spectar -xf mumble.tar 

sysinfo (Not used by SPEChpc)

sysinfo inventories various aspects of a SUT (System Under Test).
sysinfo is not used by SPEChpc since it can only gather information about the system on which 'runhpc' is invoked, which is usually just the head node of a cluster.
If useful, SPEC/HPG may consider adding support to sysinfo in order to gather system information across multiple nodes, but it may not be practical for all clusters.
For other SPEC benchmarks, such as SPEC CPU2017, it is called automatically by runcpu, but you can also run it directly from the command line.

sysinfo [-f] [-p] [-q] 

Location, platforms, and updates: The sysinfo utility is installed with SPEC CPU2017 as $SPEC/bin/sysinfo (Unix) or %SPEC%\bin\sysinfo (Windows). It supports multiple operating systems, including AIX, Linux, Mac OS X, Microsoft Windows, and Solaris. If your operating system is not supported, and if you wish to publish results, you should contact SPEC for assistance, because published results must use sysinfo.

SPEC may update the sysinfo tool from time to time, which users can download from the SPEC web site using runhpc --update. SPEC may require that published results use a specific version of the tool.

Why is sysinfo required for SPEC CPU2017 but not SPEChpc 2021?

sysinfo is very useful for SPEC CPU2017 since:

1. It creates a contemporaneous record of the system under test.
2. The record is created automatically, and is not subject to human transcription error.
3. Therefore, sysinfo can remove ambiguity such as "I remember that I upgraded the version of the operating system, but I can't quite recall whether that was before or after I did the re-run of the floating point speed tests."

However, SPEC CPU2017 is measuring a single SUT. SPEChpc 2021 is often run from a head-node and run on one or more compute nodes. Thus, sysinfo would need to be run via a scheduler on all nodes under test, gather the results, remove duplications, and consolidate the information into a single report, which could be lengthy especially if there are many different types of systems on the cluster. While technically feasible, SPEC/HPG determined, at least for the initial release, to not include this feature.

How is it called? Can I turn it off? runcpu calls sysinfo while parsing the config file. The utility is found via the setting for sysinfo_program. By default, it uses the copy in the $SPEC/bin/ directory. You could temporarily turn it off using:

sysinfo_program = 

but it must be used for results that you would like to be able to publish.

Output types: By default, information will be added to the platform notes and to some of the system description fields. To see what will be added, run it from the command line:

$ $SPEC/bin/sysinfo -q
notes_plat_sysinfo_000 = Sysinfo program /Users/somebody/spec/cpu2017/rc4/bin/sysinfo
notes_plat_sysinfo_005 = Rev: r5214 of 2017-01-15 ea4c2974a34b51078facb71b0461e013
notes_plat_sysinfo_010 = running on hostname.local Sun Jan 15 08:45:49 2017
notes_plat_sysinfo_015 =  
notes_plat_sysinfo_020 = SUT (System Under Test) info as seen by some common utilities.
notes_plat_sysinfo_025 = For more information on this section, see
notes_plat_sysinfo_030 =    https://www.spec.org/hpc2021/Docs/config.html#sysinfo
notes_plat_sysinfo_035 =  
notes_plat_sysinfo_040 = Hardware:
notes_plat_sysinfo_045 =  
notes_plat_sysinfo_050 =     Hardware Overview:
notes_plat_sysinfo_055 =  
notes_plat_sysinfo_060 =       Model Name: MacBook Pro
notes_plat_sysinfo_065 =       Model Identifier: MacBookPro11,4
.
.
.
hw_model           = MacBookPro11,4
hw_nchips          = 1
hw_ncores          = 4
hw_scache          = 256 KB per Core
hw_tcache          = 6 MB
prepared_by        = somebody  (is never output, only tags rawfile)
sw_os              = macOS 10.12.2 (16C67)
sw_other           = Xcode 8.2.1 (11766.1)
$ 
   

Editing the output: Descriptive fields (such as hw_model above) may be edited in the usual manner. The Platform Notes (notes_plat_sysinfo_NNN) cannot be edited, but you may add additional notes. Put them into a subsection with a notes tag that falls alphabetically afterwards, for example:

notes_plat_update_100 = Because of a BIOS limitation, dmidecode and sysinfo do not 
notes_plat_update_110 = report Ranks correctly for DDR4 DIMMs.  The DIMM part numbers 
notes_plat_update_120 = are reported correctly.  Therefore, please see the manufacturer 
notes_plat_update_130 = memory data sheets to determine the number of Ranks.

Comment destination: If you use -f or -p to send output to config file comments, those comments will be visible only via output_format config. They do not appear in reports, and they are not written to the config file in your $SPEC/config directory. They are written to the copy in your result directory. This can make them easy to overlook, which is why the default is to write actual notes and fields, instead of writing only comments.

Avoiding warnings about conflicting fields: You might see warnings such as:

 WARNING: Your config file sets some fields that are also set by sysinfo:
  hw_cpu_name, sw_os
   

Fields created by sysinfo (such as hw_cpu_name) are written to a section named default: at the end of your config file. The warning means that you have already set one of the listed fields in your default: section. Because the sysinfo output comes at the end of your config file, its setting will override the earlier config file settings, which might not be desirable.

To get rid of sysinfo duplicate field warnings, put your fields into any named section other than default:.

Example: Brian is using a small system for SPECspeed testing, and a large server for SPECrate testing. Brian does not like the default output from sysinfo, because it only includes the bare amount of information about memory, and does not break the memory down by DIMM types:

default:
hw_memory = 32768 MB  fixme: format is: 'n GB (n x n GB nRxn PCn-nnnnnR-n, ECC)'

Therefore, he writes the into his config file:

intspeed,fpspeed:
   hw_memory = 32 GB (2 x 16 GB 4Rx4 PC4-2133P-L)
intrate,fprate;
   hw_memory = 1 TB (32 x 32 GB 4Rx4 PC4-2133P-L)
   

After writing the above, the warnings go away, and Brian's preferred disclosure always 'wins'.

Writing your own sysinfo program: If you wish, you can write your own utility, by following the rules below. If you plan to publish your results, you should contact SPEC to ask for approval of your variant.

1. The sysinfo program can be any executable program or script. It does not have to be a Perl script.
2. If you do want to write a Perl script, you can count on specperl being on the path (as in the example above).
3. The sysinfo program is run prior to most other config file processing. This means that most variables of the form $[var] and ${var} will not work.
4. You can use $[top] to find the top of your SPEChpc 2021 tree.
5. The program should output lines of the form:

   name = value

6. The lines should be output to standard output.
7. Only informational fields (hw_*, sw_*, notes*, test_sponsor, license_num, tester, prepared_by) may be set; these settings will be inserted into the 'default:' section, and will not be able to override any per-benchmark settings.
8. Illegal lines will be silently discarded.
9. Standard config file comments are also allowed.
10. Because the line checking is fairly simple-minded, some config file features such as line continuation, block quotes, and line appending are not available.
11. The output from the sysinfo program will be included in the final config as if it had been part of the original config file.
12. Arguments can be given to the command if necessary.
13. If the sysinfo_program fails to generate any output, the tools will write a warning to the log file. Errors and warning messages from your program might not be captured to the log.

toolsver

toolsver prints information about the tools that were used by a rawfile

toolsver rawfile

SPEChpc™2021 Utilities: Copyright © 2021 Standard Performance Evaluation Corporation (SPEC)