Does a config file have to match its result?

About Downloaded Config Files and Their Corresponding Results

SPEC is frequently asked:

Q1. I downloaded a config file from www.spec.org/cpu2000, and it seems to have some differences from the corresponding performance result (as reported in the Text, HTML, PDF, or PS file). Does a SPEC CPU2000 config file have to match the result?

A1. There is one part of the config file that must strictly match; and there are parts that may be out of date. That is, the Text, HTML, PDF, or PS report may be more accurate for some information.

First, let's consider the part that must strictly match: the SPEC tools use gnu make to build the executable binaries exactly as described by the config file. If the tester changes the optimization level for 303.mumble from:

     OPTIMIZE = -O5
to
     OPTIMIZE = -O4

then the tools will force a recompile prior to running 303.mumble, in order to ensure that the config file and the binaries strictly match.

But recall (from www.spec.org/cpu2000/docs/config.html) that a config file addresses several different audiences -- one of which is the human reader. Continuing our example above, suppose that the config file contains:

    notes104  = Optimization level -O5 is used for 303.mumble

Here, we find an instance where the config file does not really match the result. We changed the OPTIMIZE line, but did not change the notes line. The config file works, the benchmark 303.mumble works, but the human reader sees erroneous information.

Therefore, SPEC allows the notes to be fixed, even after the test is run, because the notes lines do not affect the building of the binary, and do not affect how quickly it runs. If the tester changes the above line to:

    notes104 = Optimization level -O5 is not used for 303.mumble

the only effect will be to improve the accuracy of what is presented to the human reader. The tester could even add a line such as:

    notes105 = ...because 303.mumble is actually faster with -O4

after the test is complete, and both notes will be printed in the generated reports (Text, HTML, PDF, and PS).

Q2. OK, so the config file options that are addressed to the human reader can be editted after the test is complete. Does that include making changes to the testbed description?

A2. Yes. There are three common reasons why one might want to change the testbed description:

Sometimes, there are late changes to the descriptive part of a result, for example, if the manufacturer's product planning changes a product name at the last minute.
Sometimes, the SPEC review process suggests that a differently worded description would be more clear.
In addition, a tester is perfectly at liberty to (for example) take a config file which was originally developed for use on
```
    hw_model = TurboBlaster Model A4
```
and use that config file and its associated binaries on some other model. If the above line is changed to
```
    hw_model = TurboSlinger Model B5
```
the binaries, and their runtime, will not be affected.

In all three of these examples, the tester can edit the description either before the test is run, or after it is complete. Such changes have no effect on the times that were recorded.

Of course, the tester must ensure that such edits are accurate. (The tools do not attempt to enforce accuracy in testbed description items such as model name; that's up to the tester.)

Q3. How exactly are changes made after a test is run?

A3. Please bring up http://www.spec.org/cpu2000/docs/utility.html and look in the section on "rawformat".

Notice there that the changes are actually made to the rawfile; the config file itself is not changed.

Therefore, if a config file is downloaded
from www.spec.org/cpu2000, the
reader will see the config file as
it existed at the beginning of the test.

Q4. So, what's the point of posting the config file on the SPEC web site, if all these changes are allowed?

A4. First, remember that the portion of the config file that actually affects the generation of the binary is locked down. If you see in a downloaded config file that benchmark 303.mumble uses:

     OPTIMIZE = -O4

then you can be sure that 303.mumble does not use -O5. The config file provides an unambiguous record of how the binary was actually built.

Second, if you have a SPEC CPU2000 license, and you have the software environment described by the tester (in the final, formatted report - Text, HTML, PDF, or PS), you can download the config file and create binary versions of the SPEC CPU2000 benchmarks which are performance-equivalent to the binaries used by the tester. If you have the same hardware environment as described by the tester (in the final formatted report), you should be able to obtain performance similar to that which is claimed by the tester, within a small margin for run-to-run variation.

Summary

To summarize, if you download a config file from the SPEC web site, you should consider it to be the canonical record of how the binaries were actually built. But it is not the record of the system under test; if you want to know whether the binaries were run on the Model A4 or on the Model B5, look in the final, formatted report (Text, HTML, PDF, or PS).

Last updated 4-Oct-2001 jh

Standard Performance Evaluation Corporation

About Downloaded Config Files and Their Corresponding Results

Summary