The SPEChpc 2021 Benchmark Suites – A User’s Story

By Dr. Junjie Li, Research Associate, Texas Advanced Computing Center (TACC), and Nick Hagerty, HPC Engineer, Oak Ridge National Lab (ORNL)

The SPEChpc 2021 benchmark suites, a standard High-Performance Computing (HPC) test tool, proves valuable in various situations. In this blog, Nick Hagerty, HPC engineer from the Oak Ridge National Laboratory (ORNL) and Dr. Junjie Li, Research Associate from the Texas Advanced Computing Center (TACC), two of the largest HPC centers in the U.S., share their experiences with the SPEChpc 2021 Benchmark Suites in their research and daily data center operations.

Operating an academic data center that caters to diverse researchers is not easy. New hardware procurement, system validation/acceptance, and daily maintenance all require substantial performance testing. The versatility of the SPEChpc 2021 Benchmark Suites makes it the optimal tool for a multitude of tasks.

HPC procurements have become more challenging due to the rapid evolution of new hardware architectures and programming models. To provide the scientific computing community with cutting-edge computing facilities, HPC centers frequently need to evaluate emerging computing platforms. One difficulty in assessing a new platform is having suitable application benchmarks. Typical HPC applications are often optimized for specific architectures, which means that performance comparisons on novel prototype hardware are biased. Additionally, codes may not seamlessly work with different compilers and might support only a limited number of programming models, hampering their portability to new platforms. In this context, SPEC benchmarks play a significant role in complementing other applications of interest.

For example, Arm is one of the most promising new architectures for HPC. In a recently published paper titled "Application Experiences on a GPU-Accelerated Arm-based HPC Testbed", Nick and other researchers from ORNL reported their findings using the Arm testbed with Ampere ARM CPUs and NVIDIA A100 GPUs for HPC workloads. As the SPEChpc 2021 Benchmark Suites is designed for portability, minimal porting effort was required to run the benchmark on different platform, and the team was able to easily carry out testing with OpenMP offload using both NVHPC and LLVM compilers on the testbed as well as the Summit Supercomputer. They also performed SPEChpc 2021 benchmark runs with the more mature OpenACC model to evaluate the development progress of OpenMP offload compilers.

In a separate publication titled "Application Performance Analysis: A Report on the Impact of Memory Bandwidth" and presentation, Junjie and his colleagues at TACC adopted SPEChpc 2021 to evaluate the latest Intel Sapphire Rapids CPU and the impact of High-Bandwidth Memory (HBM). The well-understood performance characteristics of SPEChpc 2021 benchmark components helped the team better understand the benefits and limitations of the new hardware.

Apart from benchmarking novel hardware, the SPEChpc 2021 Benchmark Suites and other SPEC benchmarks are valuable in system validation or acceptance. The peer-reviewed result repository is a significant resource for performance studies, containing a rich amount of performance data fine-tuned by vendors. Junjie often conducts performance testing for newly installed HPC systems and newly purchased testbed hardware at TACC. By comparing performance scores with the numbers published in the SPEC result repository, he has identified many software misconfigurations and defective hardware, some of which can't be detected by simple kernel benchmarks like HPL or STREAM. When performance issues occur, it is easier to work with vendors using SPEC benchmarks than user applications because all vendors are familiar with SPEC benchmarks.

In summary, well-designed rigorous run rules, full portability across different hardware architectures, and support for various modern programming models make the SPEChpc 2021 Benchmark Suites a unique tool for HPC center operations and performance research. The SPEC High-Performance Group offers the benchmark free to academic institutes, and we encourage more HPC practitioners and researchers to explore the use of this valuable tool.

Nick Hagerty is an HPC engineer at the prestigious Oak Ridge National Lab (ORNL). ORNL is the largest US Department of Energy laboratory and the Oak Ridge Leadership Computing Facility (OLCF) at ORNL has long been the leader in HPC in the world. Currently, OLCF operates the world’s fastest supercomputer, Frontier, and the former fastest supercomputer, Summit.

Dr. Junjie Li is a Research Associate at the Texas Advanced Computing Center (TACC) and the University of Texas at Austin. TACC is a leadership class computing center funded by the National Science Foundation (NSF) and is the largest pure academic HPC center in the U.S., operating several medium to large-scale supercomputers including Frontera, Stampede3, Vista, and Lonestar6.

[Back to SPEC blog]