Last updated: 17 Feb 2001 3pm wj
(To check for possible updates to this document, please see
http://www.spec.org/hpg/omp2001
)
Everyone who uses SPEC OMP2001 needs runspec. It is the primary tool in the suite. It is used to build the benchmarks, run them, and report on their results. All users of OMP2001 should read this document.
In order to use runspec, you need a "config file", which contains detailed instructions about how to build and run the benchmarks. You may not have to learn much about config files in order to get started. Typically, you start off using a config file that someone else has previously written.
Where can you find such a config file? There are 2 primary sources:
$SPEC/config (Unix) or %SPEC%\config
(NT). You may find that there is a already a config file there
with a name that indicates that it is appropriate for your system. You
may even find that "default.cfg" already contains settings that
would be a good starting place for your system.Alternatively, you can write your own, using the instructions in
$SPEC/docs/config.html(Unix) or%SPEC%\docs.nt\config.html(NT)
and the examples in
$SPEC/docs/*.cfg(Unix) or%SPEC%\docs.nt\*.cfg(NT).
The SPEC tools have followed two design principles regarding defaults:
This means (the good news) that something sensible will usually happen, even when you aren't explicit about what you want. But it also means (the bad news) that if something unexpected happens, you may have to look in several places in order to figure out why it behaves differently than you expect.
The order of precedence for settings is:
| Highest precedence: | runspec command |
|---|---|
| Middle: |
config file |
Lowest: |
the tools as shipped by SPEC |
Therefore, when this document tells you that something is the default, bear in mind that your config file may have changed that setting. If in doubt, check the config file.
The structure of the OMP2001 directory tree is:
$SPEC or %SPEC% - the root directory
benchspec
OMPM2001 - medium OpenMP benchmarks
OMPL2001 - large OpenMP benchmarks
bin - tools to run and report on the suite
config - config files
result - log files and reports
tools - sources for the OMP2001 tools
Within each of the individual benchmarks, the structure is:
nnn.benchmark - root for this benchmark
Spec - SPEC metadata about the benchmark
data
all - data used by all runs (if needed by the benchmark)
ref - the real data set, required for all result reporting
test - data for a simple test that an executable is functional
train - data for feedback-directed optimization
exe - compiled versions of the benchmark
run - all builds and runs take place here
src - the sources for the benchmark
When you find yourself wondering "Where did all my disk space go?", the answer is "The run directories." All build and run activity takes place in numbered run directories - e.g.
$SPEC/benchspec/OMPM2001/310.wupwise_m/run/00000001
To get your disk space back, see the documentation of the various cleaning options, below; or issue a command such as the following (on Unix systems; NT users can select the files with Explorer):
rm -Rf $SPEC/benchspec/C*/*/run
Because all the build and run activity takes place in separately created directories, the tools are able to provide limited support for multiple users of the same directory tree:
The support for multiple users is subject to some limitations:
config/phil.cfg
or benchspec/OMPM2001/310.wupwise_m/exe/wupwise_m_base.les
If these limitations cause concern, you should give each user their own copy of the entire SPEC OMP2001 directory tree, and use your operating system's file protection scheme.
Before using runspec, you need to:
more /cdrom/docs/install_guide_unix.htmlor
notepad E:\docs.nt\install_guide_nt.html
3.a. If you are using a Unix system:
Change your current directory to the top-level SPEC directory and!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!! START A BOURNE-COMPATIBLE SHELL !!!!!!!!!!!!! ! (if you are not already in one) ! ! ! ! That is, you need to use a command interpreter that ! ! supports the syntax used by shrc, which tries to be ! ! highly portable by avoiding proprietary or uncommon ! ! shell features. Many shells -- such as ash, bash, ! ! ksh, and zsh -- provide some degree of compatibility ! ! with the Bourne shell. The C shell (csh) definitely ! ! is NOT compatible with Bourne shell syntax. If you ! ! are in doubt, or if you see unexpected error messages ! ! from shrc, you can usually start a Bourne shell by ! ! typing 'sh'. ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Then, source the file shrc. If you prefer some other shell, you can probably spawn it after sourcing shrc; when you do so, please be sure that your new shell does not clear the path variables set up by shrc!
For example:
% cd /usr/rahul/omp2001 % sh $ . ./shrc <- that's dot-space-dot-slash-shrc $ csh %3.b. If you are using a Windows/NT system:
Read the comments to the file
shrc.batand make the appropriate edits for your compiler paths. Caution: you may find that the lines are not correctly formatted (the text appears to be all run together) when you edit this file. If so, see the section "Using Text Files on NT" in install_guide_nt.html
Then set the path using your edited shrc.bat, for example:
F:\bodo\omp2001> shrc+-------------------------------------------------------------------+ | WARNING: Under both Unix and NT, you will typically want to make | | sure that the environment variable SPEC is NOT defined when you | | invoke shrc. If it is already defined (e.g. from a run of some | | other SPEC benchmark suite), you should undefine it first, e.g. | | by logging out and logging in, or by using 'unset'. | +-------------------------------------------------------------------+
Runspec is easiest to use when:
In this lucky circumstance, all that needs to be done is to name the config file, select which benchmark suite is to be run: "medium" for OMPM2001 or "large" for OMPL2001, and say '--reportable' if one wants to attempt a full run that complies with the SPEC OMP2001 run rules.
For example, suppose that Wilfried has given Ryan a config file with some new integer optimizations for a Unix system. Ryan could say:
% cd /usr/ryan/omp2001
% sh
$ . ./shrc
$ runspec --config wilfried_new.cfg --reportable medium
As a another example, suppose that Reinhold has given Kaivalya an NT config file with changes from 12 August, and Kaivalya wants to run the floating point suite. He might say something like this:
F:\kaivalya\omp2001> shrc
F:\kaivalya\omp2001" runspec --config reinhold_aug12a --reportable large
If you want to run a subset of the benchmarks, rather than running the whole suite, you can name them. Since a reportable run uses an entire suite, you'll need to turn off reportable:
% runspec --config arrian_dec25j.cfg --noreportable 312.swim_m 330.art_m
Look for the output of your runspec command in the directory $SPEC/result (Unix) or %SPEC%\result (NT). There, you will find log files and result files. More information about log files can be found in config.html.
The format of the result files depends on what was selected in your config file, but will typically include at least *.asc for ASCII text, and will always include .raw, for raw (unformatted) run data. More information about result formats can be found below, under --output_format. Note that you can always re-generate the output, using the --rawformat option, also documented below.
If simple commands such as the above don't seem to do what you like, then you'll have to keep reading this document.
The syntax for the runspec command is:
runspec [options] [list of benchmarks to run]
Options are described in the following sections. There, you you will notice that many options have both long and short names. The long names are invoked using two dashes, and the short names use only a single dash. For long names that take a parameter, you can optionally use an equals sign. Long names can also be abbreviated, provided that you still enter enough letters for uniqueness. For example, the following commands all do the same thing:
runspec --config=dianne_july25a --debug=99 medium
runspec --config dianne_july25a --debug 99 medium
runspec --conf dianne_july25a --deb 99 medium
runspec -c dianne_july25a -v 99 medium
The list of benchmarks to run can be either an entire suite ("int" or "medium") or one or more individual benchmarks. Individual benchmarks can be named, numbered, or both; and they can be abbreviated, as long as you enter enough characters for uniqueness. For example, each of the following commands does the same thing:
runspec -c jason_july09d --noreportable 314.mgrid_m 310.wupwise_m
runspec -c jason_july09d --noreportable 314 310
runspec -c jason_july09d --noreportable mgrid_m wupwise_m
It is also possible to exclude a benchmark, using a hat (^, also known as carat, typically found as shift-6). For example, suppose your system lacks a C++ compiler, and you therefore cannot run the benchmark 252.eon. You could run all of the integer benchmarks except eon by entering a command such as this one:
runspec --noreportable -c kathy_sep14c large ^art_m
Note that if hat has significance to your shell, you may need to protect it from interpretation by the shell, for example by putting it in single quotes. On NT, you will need to use both a hat and two quotes for each benchmark you want to exclude, like this:
E:\omp2001" runspec --noreportable -c debbie_apr26a large "^wupwise_m"
"^art_m" Every time runspec is used, it normally takes some kind of action for the set of benchmarks specified at the end of the command line (or defaulted from the config file). The default action is "validate", which means that the benchmarks will be built if necessary, the run directories will be setup, the benchmarks will be run, and reports will be generated.
If you want to force a different action, then you can enter one of the following runspec options:
--action build |
Compile the benchmarks. More information about compiling may be found in config.html, including information about additional files that are output during a build. |
--action run |
Run the benchmarks but do not bother to verify that they got the correct answers. This option is not normally useful, but can be selected if, for example, you are generating a performance trace and wish to avoid tracing some of the tools overhead. |
--action setup |
Setup the run directories. Copy executables and data to work directories. |
--action validate |
Build (if needed), run, and generate reports |
In addition, the following cleanup actions are available (in order by level of vigor):
--action clean |
Empty all run and build directories for the specified benchmark set. For example, if user Jeff enters the command: runspec --action clean --config may12a medium the tools would remove run directories for the floating point benchmarks which are tagged as belonging to Jeff and which contain benchmarks generated by config file may12a.cfg. |
--action trash |
Same as clean, but do it for all users of this SPEC directory tree. |
--action realclean |
A synonym for --action trash |
--action clobber |
Clean + remove all executables of the current type |
--action nuke |
Remove everybody's run and build directories and all executables |
Alternative cleaning method:
If you prefer, you can clean disk space by entering commands such as the following (on Unix systems):
rm -Rf $SPEC/benchspec/C*/*/run
rm -Rf $SPEC/benchspec/C*/*/exe
NT users can achieve a similar effect using Windows NT Explorer. Notice that the above commands not only empty the contents of the run and exe directories; they also delete the directories themselves. That's fine; the tools will re-create the run and exe directories if they are needed again later on.
Most users of runspec will want to become familiar with the following options.
--action <action>-a <action> --config <name>-c <name> --help -h, -? --ignore_errors -I, --ignoreerror --iterations <number> -n <number> --output_format <format>-o <format> --rawformat <rawfiles>-R <rawfile> runspec
--rawformat --output_format html,ps \ result/OMPM2001.324.raw You
can achieve the same effect by using rawformat: rawformat --output_format
html,ps result/OMPM2001.324.raw For more information about rawformat,
please see $SPEC/docs/utility.html (Unix) or %SPEC%\docs.nt\utility.html
(NT)--rebuild -D --reportable-s, --strict, --noloose iterations >=2.
--noreportable -l, --nostrict, --loose runspec --config golden --iterations 1 art_m as the SPEC
tools will inform you that you cannot change the number of iterations
on a reportable run. But the following command will override the config
file and just run eon once: runspec --config golden --iterations
1 --noreportable art_m --tune <tuning>--tuning <tuning>, -T <tuning>
--deletework -d --extension <name>--ext <name>, -e <name>
--machine <name> --mach <name>, -m <name> --make_no_clobber -C $SPEC/docs/utility.html (Unix) or %SPEC%\docs.nt\utility.html
(NT)--setprocgroup--setprocgroup --nosetprocgroup --setprocgroup --size <size> -i <size>, --input <size>
--table --table --notable --table --unbuffer-f --unbuffer.--verbose <n>--debug <n>, -v <n> --version-V