SPECpower® Committee

The increasing demand for energy-efficient IT equipment has resulted in the need for power and performance benchmarks. In response, the SPEC community established the SPECpower Committee, an initiative to augment existing or create new industry-standard benchmarks and tools with a power/energy measurement. Leading engineers and scientists in the fields of benchmark development and energy efficiency, representing a diverse and competitive set of companies, made a commitment to tackle this task. The development of the first industry-standard benchmark that measures the power and performance characteristics of server-class compute equipment began on January 26 2006.

The SPECpower Committee includes representatives from AMD, Dell, HPE, IBM, Intel, Microsoft, and the University of Würzburg. Veteran SPEC benchmark developers Hansfried Block and Greg Darnell are supporting contributors.Meet the team.

SPECpower Committee timeline, starting with foundation in 2006 and including benchmark/tool releases.

The SPEC PTDaemon Interface

The power temperature daemon (also known as the PTDaemon interface) is used to offload the work of controlling a power analyzer or temperature sensor during measurement intervals to a system other than the SUT. It hides the details of different power analyzer interface protocols and behaviors from the benchmark software, presenting a common TCP/IP-based interface that can be readily integrated into different benchmark harnesses. Products already using the PTDaemon interface include SPECpower_ssj®2008, SPECweb®2009, and SPECvirt_sc®2013, the SERT® suite, and the Chauffeur® WDK.

SPECpower_ssj2008

In December of 2007, the SPECpower_ssj2008 benchmark was released, which exercises the CPUs, caches, memory hierarchy, and the scalability of shared memory processors on multiple load-levels. The benchmark runs on a wide variety of operating systems and hardware architectures. In version 1.10, which was released on April 15 2009, SPEC augmented SPECpower_ssj2008 with multi-node support (e.g., blade-support). Several enhancements and code changes to all benchmark components, documentation updates, and run and reporting rules enhancements were included in version 1.11, released September 13, 2011 and version 1.12, released March 30, 2012.

The Committee is working on the next SPECpower benchmark, designed for the next generation of servers. This is based on the Chauffeur test harness, supported by the latest PTDaemon interface. The major effort is currently focused on designing a new transactional workload supporting multiple load levels. Under consideration is native code support, GPGPU enablement, extended storage support and targeting a run time of not more than two hours.

The SERT Suite

The work on the SERT suite started around 2009 with the ground-breaking design of the Chauffeur framework, followed by the design and implementation of the workload, automated hardware and software discovery, and GUI. After a series of successful Alpha and Beta test phases, the SERT suite was released on February 26, 2013. On March 15, 2013, the United States Department of Energy's (DOE) Environmental Protection Agency (EPA) mandated the reporting of SERT results in their ENERGY STAR for Server Certification program. The collection of these data sets was the basis for formulation of thresholds for future energy programs.

The SPECpower Committee released four upgrades over the years to further enhance the usability of the SERT 1 Suite and to increase its platform support.

The primary focus of the SERT 2 Suite was the reduction of its run time and improvements on the automated hardware and software discovery to reduce test time and cost. In addition, the support of additional power analyzers and interfaces was adopted in order to provide wider choices for users. Also, for more accurate characterization of memory, some major improvements of the memory subsystem worklets, were implemented. Simultaneously, the SPECpower Committee, in collaboration with the more academic-focused SPEC Power Research Group, conducted large-scale experiments in order to determine the server efficiency metric for the SERT 2 Suite which enabled its release on March 28, 2017.

A significant milestone was achieved on September 17, 2018 as the U.S. EPA released the final version of the ENERGY STAR Version 3.0 Computer Servers Program Requirements. It defines Active State Efficiency Thresholds based on the SERT 2.0.1 metric, replacing the idle power thresholds and will determine ENERGY STAR eligibility effective June 17, 2019.

March 15, 2019: The European Union adopts regulation utilizing the SERT suite's active and idle state efficiency metrics. The Official Journal of the European Union indicated that member states elected to adoptserver and data storage product Lot9 Ecodesign Regulation (EU) 2019/424. This regulation went into effect March 1st, 2020, with the SERT 2 suite being the metric of choice for active and idle state server efficiency calculations on products within the scopeof this regulation.

March 29th, 2019: Japan METI Top Runner Program announces regulation utilizing the SERT suite. In a memo published March 29th, 2019, Japan's Ministry of Economy, Trade, and Industry (METI) indicated they had accepted a shift in computer server energy conservation standard that takes into account the wholeserver rather than simply the CPU. The SERT suite version 2 is the metric chosen by METI to demonstrate performance per watt for computer servers in this geography.

December 6, 2019: SPEC's SERT suite 2.0.3 Released. This version includes multiple enhancements including support for Marvell ThunderX2 processors, support for HP-UX on Itanium, and a fix for IBM Power Large Memory configurations on Linux.

November 2, 2020: ISO/IEC 21836:2020 adopts SPEC's SERT suite for measuring server efficiency. "ISO/IEC 21836:2020 Information technology, Data centres, Server energy effectiveness metric" has been published and the SERT suite has been chosen to measure server efficiency. Please see "ISO/IEC 21836:2020 SERTSuite Requirements" for guidance on using SERT suite with this ISO standard.

June 15, 2021: SPEC releases a new ISO-compliant version of the SERT suite, which forms the core of the ISO/IEC 21836:2020 standard. This next-generation measurement standard was designed in collaboration with leaders of global energy-efficiency regulatory programs. The SERT 2.0.4 Suite supports certification of the latest ARM, Power, and x86 architectures driving global regulatory energy goals.

January 1, 2022: As of the first of January, development of the SERT Suite has transitioned to SPEC Internation Standards Group(ISG).

The Chauffeur Worklet Development Kit (WDK)

The Chauffeur framework from SERT is made available as the Chauffeur Worklet Development Kit (WDK). This kit can be used to develop new workloads (or "worklets" in Chauffeur terminology). Researchers can also use the WDK to configure worklets to run in different ways, in order to mimic the behavior of different types of applications. These features can be used in the development and assessment of new technologies such as power management capabilities. The first version of the Chauffeur WDK was released on May 18, 2014, and succeeded by the 1.1.0 release on May 26, 2015 and version 2.0 on April 18, 2017.

The Chauffeur WDK 2.0 further simplifies workload development for researchers analyzing computer performance and power consumption. Targeting both industry and academia, the WDK provides the infrastructure for SERT 2.0, providing run-time & platform configuration support to simplify worklet development. This major release includes significant enhancements to the hardware detection, customization options of generating HTML reports, and developer documentation. It has now reduced memory requirements for the Director when signing results files and reduced the size of the result output for large systems or clusters. Chauffeur WDK 2.0 added worklet-specific normalization of results and an updated list of supported operating systems and the latest PTDaemon integration for power analyzers and temperature sensors, along with data collection, validation and reporting.

The Power and Performance Benchmark Methodology

The Power and Performance Benchmark Methodology document is intended for performance benchmark designers and implementers who want to integrate a power component into their benchmark. The document may also serve as an introduction to those who need to understand the relationship between power and performance metrics in computer systems benchmarks. The assumption is that the business model and benchmark application are already selected and may already be implemented. Guidance is provided for including power metrics in existing benchmarks, as well as altering existing benchmarks and designing new ones to provide a more complete view of energy consumption. Benchmarks and tools already using this methodology document include SPECpower_ssj2008, SPECweb2009, SPECvirt_sc2013, the SERT suite, and the Chauffeur WDK.

What's next for SPECpower Committee?

The SPECpower committee is working on a successor to the popular (and still the only) power and performance benchmark, the SPECpower_ssj2008 benchmark. The new benchmark is based on the SPECpower Committee's expertise in the field of energy assessment across a wide spectrum of server configurations. It will have extensive Full Disclosure Report (FDR) description, include automatic discovery of hardware / software described in FDR, with automatic update support mechanism in place. It will also include tests to verify compliance with run and reporting rules and detect result tampering. It will be economical, easy to use - requiring no specific skills or equipment.

The key characteristics under consideration for the new benchmark include assessment of new server components (e.g., GPGPU), and inclusion of emergent cypher support with stronger key length (AES-256, SHA2-512, RSA 4096 bit, ECDSA P-384 etc.). The benchmark is being developed in collaboration with academia, and will stress multiple subsystems in parallel (multi-dimensional), and emergent technologies like AI, ML, and NLP are being explored.

SPEC and the names SERT, SPECpower, Chauffeur, and PTDaemon are trademarks of the Standard Performance Evaluation Corporation. Additional product and service names mentioned herein may be the trademarks of their respective owners. Copyright © 1988-2020 Standard Performance Evaluation Corporation (SPEC). All rights reserved.