Searching for Workloads — Help Shape the Next SPEC CPU Benchmark Suites

By James Bucek, SPEC CPU Chair

If you know of an application that needs a CPU benchmark, you may earn a cash reward!

SPEC exists to develop benchmarks that vendors, businesses, and individuals can trust to make key purchasing decisions based on fair comparisons between different systems or solutions. One way we do this is bringing together representatives from a variety of companies, including competitors, to design benchmarks that fairly represent real-world workloads.

For SPEC CPU, an equally important aspect of creating better benchmarks is that we base them on actual applications or workloads in use today, not just representative workloads. This separates SPEC CPU from many of the common micro-benchmarks and gives a user more confidence in the workloads behind the benchmark, as well as greater assurance that the results produced are standard and reproducible, allowing for more accurate comparisons between solutions.

To that end, our SPEC CPU® v8 Benchmark Search Program remains open (through February 24, 2022), and we are inviting industry and academic representatives, who are currently not members of SPEC, to suggest applications or workloads they would like to see used in the next CPU-intensive benchmark suite, which is currently under development within SPEC and designated as the SPEC CPU v8 benchmark.

What’s in it for you? Possibly a cash reward. For each step of the process that a submission passes, SPEC will compensate the program submitter as much as $3000, though step prizes vary by each level. A submission that passes all the steps and is included in the next SPEC CPU benchmark suite can net the program submitter up to a total of $9000, as well as a license for the new benchmark suite when released. (Note, not all submissions are guaranteed to pass the first step; and SPEC, in its sole judgement, may decide at any time to discontinue work on a submission.)

What’s in it for SPEC? Our benchmark search program expands the computing environments we are exposed to and broadens the pool of possible applications to include in our benchmark suites. This helps us better represent work being done by individuals, companies and institutions beyond the scope of current SPEC members.

Things to consider

SPEC CPU benchmarks provide performance metrics for comparing systems on a known compute-intensive workload, with emphasis on the system's processor(s), memory hierarchy, and compiler(s). To ensure the next SPEC CPU benchmark suite continues to do this, we are looking for benchmark application or workload candidates that are:

• Used by real users.
• Compute bound, or can have its compute bound portion excerpted.
• Portable or can be ported to multiple hardware architectures and operating systems with reasonable effort.
• State of the art for the given field.
• Derived from a representative application.
• Capable of solving problems of varying sizes. The SPEC CPU 2017 suite used up to 4 workloads, in various capacities, for its benchmarks.
• Reasonably predictable as to its code path. For example, minor differences in floating point accuracy across platforms should not cause the application to do wildly different work.

The benchmark candidate can come from any area of computing. The current SPEC CPU 2017 benchmark suite includes applications from more than 20 areas, from interpreters and compilers to explosion modeling and fluid dynamics, to 3D rendering and animation, to computational electromagnetics.

How to participate

The SPEC CPU® v8 Benchmark Search Program runs through February 24, 2022. If you’re interested in participating, please read the complete program description, which includes the official rules, key limitations, and how to submit an application.

Submitting an application isn’t just a way to improve your industry and perhaps earn some money. It’s also a way to get to know SPEC and our mission. As an organization, SPEC has worked diligently for nearly 35 years to make computing better, enabling vendors to understand how to improve their products and allowing purchasers to make better decisions about the products that will meet in their needs. And as anyone who has been a member of SPEC for a while will attest, we enjoy working together for a common cause, and we are always looking for new members.

[Back to SPEC blog]